Image data processing method, device, equipment, storage medium and program product

Abstract

The disclosed embodiments disclose an image data processing method, apparatus, device, storage medium and program product, wherein the image data processing method comprises: for image data of any application scenario, inputting the first image data of the current round into a controller, and outputting a plurality of first transformation operations matching the first image data; transforming the first image data based on the first transformation operation to obtain the transformed first image data; inputting the transformed first image data into an image segmentation model, and outputting a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located. In this way, the image segmentation model can be applied to various application scenarios without any fine-tuning of the model parameters of the image segmentation model, and has a wide range of applications.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the field of image processing technology, and in particular to an image data processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Basic segmentation models such as the Segment Anything Model (SAM) and the Semantic-Enhanced Interactive Segmentation Model (SEEM) can identify and segment objects in natural images. However, in multiple application scenarios such as smart healthcare, industrial manufacturing, and intelligent transportation, the performance of basic segmentation models will drop significantly, and the segmentation accuracy is low. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure at least provide an image data processing method, apparatus, device, storage medium and program product.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] On the one hand, an embodiment of the present disclosure provides an image data processing method, which includes: for image data of any application scenario, inputting the first image data of the current round into a controller, and outputting multiple first transformation operations matching the first image data; the multiple first transformation operations include multiple identical or different transformation operations; the multiple identical transformation operations include multiple identical transformation actions, but the multiple transformation factors corresponding to each transformation action are different; transforming the first image data based on the first transformation operation to obtain transformed first image data; inputting the transformed first image data into an image segmentation model, and outputting a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located.

[0006] It can be understood that for the image data of any application scenario, the controller first obtains multiple first transformation operations (optimal transformation operations) that match the first image data of the current round, and then transforms the first image data through the optimal transformation operation, and then uses the image segmentation model to segment the transformed first image data; in this way, no matter which application scenario the image data is, the controller can obtain the optimal transformation operation, and the image data is transformed through the transformation operation and then input into the image segmentation model for segmentation, which can significantly improve the segmentation accuracy of the image segmentation model. Since the image segmentation model processes the image data after the transformation using the optimal image prompt, the image segmentation model can achieve the expected performance using less image data, reducing the computer resources, data set size and training time required for training. The image data processing method provided by the disclosed embodiment does not require any fine-tuning of the model parameters of the image segmentation model, so that the image segmentation model can be applied to various application scenarios and has a wide range of applications.

[0007] In some embodiments, the above-mentioned inputting the first image data of the current round into the controller and outputting multiple first transformation operations matching the first image data includes: inputting the first image data into multiple prediction heads in the controller, outputting multiple first transformation actions matching the first image data and the transformation strength of each first transformation action; the first transformation operation includes the first transformation action and the transformation strength of each first transformation action.

[0008] It can be understood that, through multiple prediction heads, it is possible to learn image features in the first image data from different angles, and output multiple first transformation actions matching the first image data and the transformation strength of each first transformation action.

[0009] In some embodiments, the above-mentioned input of the first image data to multiple prediction heads in the controller, outputting multiple first transformation actions matching the first image data and the transformation strength of each first transformation action, includes: mapping the first image data to the vector space to obtain the query embedding of the first image data; inputting the query embedding to multiple prediction heads, and outputting the prediction results of multiple prediction heads; the prediction results are multiple transformation actions matching the first image data and the transformation strength of each matching transformation action obtained by each prediction head; and integrating the prediction results of multiple prediction heads to obtain the first transformation actions and the transformation strength of each first transformation action.

[0010] It can be understood that by first performing predictions through multiple prediction heads and then integrating the prediction results of the multiple prediction heads, a transformation action and transformation strength that better matches the first image data can be obtained, thereby improving the segmentation accuracy of the image segmentation model.

[0011] In some embodiments, the above-mentioned inputting the query embedding into multiple prediction heads and outputting the prediction results of multiple prediction heads includes: inputting the query embedding into the recurrent neural network of each prediction head respectively, and outputting the operation embedding corresponding to the query embedding; inputting the operation embedding corresponding to the query embedding into the feedforward neural network of each prediction head, and outputting the prediction result of each prediction head.

[0012] It can be understood that through the recurrent neural network, the operation embedding corresponding to the query embedding can be obtained; through the feedforward neural network, the prediction result of the prediction head can be obtained.

[0013] In some embodiments, the image data processing method further includes: determining a transformation intensity of the transformation action based on a property of the transformation action.

[0014] In some embodiments, the above-mentioned determination of the transformation strength of the transformation action based on the attributes of the transformation action includes: when the transformation action is Gaussian blur, determining the transformation strength of the Gaussian blur based on the Gaussian kernel size and target parameters of the Gaussian blur; the target parameters characterize the degree to which any pixel in the image data is affected by the neighboring pixels during Gaussian blur; when the transformation action is a linear transformation, determining the transformation strength of the linear transformation based on the scaling factor and offset of the linear transformation; when the transformation action is a brightness transformation, determining the transformation strength of the brightness transformation based on the adjustment range of the brightness transformation; when the transformation action is a saturation transformation, determining the transformation strength of the saturation transformation based on the adjustment range of the saturation transformation.

[0015] In some embodiments, the image data processing method also includes: adjusting the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain an adjusted controller; the real mask is the area where the target object is located in the preset first image data; inputting the second image data of the next round into the adjusted controller, outputting multiple second transformation operations matching the second image data, transforming the second image data based on the second transformation operations, and inputting the transformed second image data into the image segmentation model, outputting the predicted mask of the second image data, and continuing to adjust the model parameters of the controller based on the similarity between the predicted mask of the second image data and the real mask of the second image data until a preset stop condition is reached.

[0016] It can be understood that by adjusting the model parameters of the controller according to the similarity between the predicted mask of the first image data and the real mask of the first image data, the controller can continuously adapt to image data of different application scenarios.

[0017] In some embodiments, the above-mentioned method of adjusting the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller includes: determining the information entropy of the output probability of the first transformation operation; adjusting the model parameters of the controller based on the information entropy and the similarity to obtain the adjusted controller.

[0018] It can be understood that adjusting the model parameters of the controller according to the information entropy of the output probability of the first transformation operation and the similarity between the predicted mask of the first image data and the real mask of the first image data can enable the controller to continuously adapt to image data of different application scenarios.

[0019] In some embodiments, the above-mentioned determination of the information entropy of the output probability of the first transformation operation includes: determining a loss value based on the similarity degree and the target loss function; determining the information entropy based on the loss value; and the loss value is positively correlated with the information entropy.

[0020] It can be understood that by adjusting the information entropy of the output probability of the transformation operation based on the loss value, the randomness of the transformation operation is adjusted according to the segmentation accuracy of the model.

[0021] In some embodiments, the above-mentioned adjusting the model parameters of the controller based on information entropy and similarity to obtain an adjusted controller includes: determining the loss value based on the similarity and the target loss function; using a gradient descent algorithm to adjust the model parameters of the controller based on the loss value and information entropy to obtain the adjusted controller.

[0022] It can be understood that adjusting the model parameters of the controller according to the loss value and information entropy can reduce the non-differentiability of the sampling operation.

[0023] In some embodiments, the above-mentioned determination of the loss value based on the similarity degree and the target loss function includes: determining the intersection-and-union ratio between the predicted mask of the first image data and the true mask of the first image data; the intersection-and-union ratio characterizes the similarity between the predicted mask of the first image data and the true mask of the first image data; and determining the loss value based on the target loss function and the intersection-and-union ratio.

[0024] It can be understood that determining the loss value according to the intersection-and-union ratio between the predicted mask of the first image data and the real mask of the first image data can more accurately measure the difference between the predicted mask and the real mask.

[0025] In some embodiments, the image data processing method also includes: determining the mask quality of the first transformation operation based on the similarity between the predicted mask of the first image data and the actual mask of the first image data; when the mask quality of the first transformation operation is higher than the mask quality of the pre-stored target transformation operation, updating the target transformation operation to the first transformation operation; and transforming the second image data of the next round based on the updated target transformation operation.

[0026] It can be understood that the transformation operation with a higher IOR value is retained so that the transformation operation with a higher IOR value can be directly used to transform the image in an actual application scenario.

[0027] On the other hand, an embodiment of the present disclosure provides an image data processing device, which includes: a matching module, configured to input the first image data of the current round into a controller for image data of any application scenario, and output multiple first transformation operations matching the first image data; the multiple first transformation operations include multiple identical or different transformation operations; the multiple identical transformation operations include multiple identical transformation actions, but the multiple transformation factors corresponding to each transformation action are different; a transformation module, configured to transform the first image data based on the first transformation operation to obtain the transformed first image data; a prediction module, configured to input the transformed first image data into an image segmentation model, and output a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located.

[0028] On the other hand, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0029] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.

[0030] On the other hand, an embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0031] On the other hand, an embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented.

[0032] In the disclosed embodiment, for the image data of any application scenario, a plurality of first transformation operations (optimal transformation operations) matching the first image data of the current round are first obtained through the controller, and then the first image data is transformed through the optimal transformation operation, and then the transformed first image data is segmented by the image segmentation model; in this way, no matter which application scenario the image data is, the optimal transformation operation can be obtained through the controller, and the image data is transformed through the transformation operation and then input into the image segmentation model for segmentation, which can significantly improve the segmentation accuracy of the image segmentation model. Since the image segmentation model processes the image data after the transformation using the optimal image prompt, the image segmentation model can achieve the expected performance by using less image data, which reduces the computer resources, data set size and training time required for training. The image data processing method provided by the disclosed embodiment does not require any fine-tuning of the model parameters of the image segmentation model, so that the image segmentation model can be applied to various application scenarios and has a wide range of applications.

[0033] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0035] Figure 1 A schematic diagram of an implementation process of an image data processing method provided in an embodiment of the present disclosure Figure 1 ;

[0036] Figure 2 A schematic diagram of an implementation process of an image data processing method provided in an embodiment of the present disclosure Figure 2 ;

[0037] Figure 3 A schematic diagram of an implementation process of an image data processing method provided in an embodiment of the present disclosure Figure 3 ;

[0038] Figure 4 A schematic diagram of an implementation process of an image data processing method provided in an embodiment of the present disclosure Figure 4 ;

[0039] Figure 5 A schematic diagram of the structure of an image data processing device provided by an embodiment of the present disclosure;

[0040] Figure 6 A hardware entity diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0045] The present disclosure provides an image data processing method, which can be executed by a processor of a computer device. The computer device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or other device with data processing capabilities. Figure 1 As shown, the method includes the following steps 101 to 103:

[0046] Step 101: For image data of any application scenario, first image data of the current round is input to a controller, and a plurality of first transformation operations matching the first image data are output.

[0047] The plurality of first transformation operations include a plurality of identical or different transformation operations; the plurality of identical transformation operations include a plurality of identical transformation actions, but the plurality of transformation factors corresponding to each of the transformation actions are different.

[0048] Application scenarios include, but are not limited to: face recognition, unmanned driving, remote sensing image recognition, robot intelligent positioning, industrial product quality inspection, etc. Image data includes the image itself, the attribute information of the image, and the real mask of the image. The image itself includes the pixel values ​​of the image and the arrangement of the pixel values. The attribute information of the image includes, but is not limited to: the source, size, color space, bit depth, metadata, compression information, transparency channel, image format and other information of the image.

[0049] Image data in different application scenarios have different image features. For example, in the application scenario of face recognition, the image data is mainly the facial image of the person; in the application scenario of unmanned driving, the image data is mainly the real-life image of the vehicle and traffic roads; in the application scenario of industrial product quality inspection, the image data is the captured image of the product.

[0050] Example 1: The application scenario of industrial product quality inspection may be the scenario of welding battery cell poles. In this case, the image data is a captured image of the battery cell pole. The image data processing method provided in the embodiment of the present disclosure is used to determine the edge of the pole included in the captured image so as to accurately weld the pole based on the edge of the pole.

[0051] Example 2: The application scenario of industrial product quality inspection can be the winding of the tab; in this case, the image data is a side-shot image of the tab winding. The image data processing method provided by the embodiment of the present disclosure is used to determine the number of layers of the tab winding to avoid the tab folding and contacting the battery cell body to cause a short circuit. In addition, the image data processing method provided by the embodiment of the present disclosure can also be applied to other scenarios such as the appearance inspection of battery packs / bare cells.

[0052] The controller is a model that predicts the transformation operations that match the image data. In one possible implementation, the basic structure of the controller can be a long short-term memory network (LSTM). Each prediction head of the controller is a linear layer that can predict the transformation operation.

[0053] One round of training refers to a complete processing of the image data by the model, and one round can also be called a batch. Transformation operations, also known as image transformation strategies, image prompts, etc., are used to enhance image data to improve the segmentation accuracy of the image segmentation model. Transformation operations can include transformation actions and transformation strengths. Transformation actions can include but are not limited to: Gaussian blur, linear transformation, brightness, and saturation transformation, etc., and the transformation strength can be divided into 0-10 levels.

[0054] The first image data refers to the image data of the current round. The first transformation operation refers to multiple transformation operations that match the first image data, that is, the first transformation operation may refer to the optimal transformation operation combination sampled. Of course, the first transformation operation may also refer to a transformation operation that matches the first image data.

[0055] The multiple transformation operations matched with the first image data may be multiple identical transformation operations, which may include multiple identical transformation actions, but each transformation action corresponds to a different transformation factor, wherein the transformation factor can reflect the transformation strength of the transformation operation.

[0056] For example, the transformation action may be a linear transformation, and the transformation factor of the linear transformation may include a scaling factor and an offset. In this case, the multiple transformation operations matched with the first image data refer to linear transformations with different scaling factors and offsets.

[0057] As another example, the transformation action may be Gaussian blur, and the transformation factor of Gaussian blur may include the Gaussian kernel size and a parameter characterizing the degree to which any pixel in the image data is affected by neighboring pixels during Gaussian blur. In this case, the multiple transformation operations matching the first image data refer to linear transformations with different Gaussian kernel sizes and parameters.

[0058] In some embodiments, the specific implementation method of step 101 may be: inputting the first image data into multiple prediction heads in the controller, outputting multiple first transformation actions matching the first image data and the transformation strength of each of the first transformation actions; the first transformation operation includes the first transformation action and the transformation strength of each of the first transformation actions.

[0059] The first transformation action refers to a plurality of transformation actions matched with the first image data. The plurality of transformation actions may be the same or different; and the transformation strengths of the plurality of transformation actions may be the same or different.

[0060] Step 102: transform the first image data based on the first transformation operation to obtain the transformed first image data.

[0061] In some implementations, step 102 may be specifically implemented by performing a transformation of the first image data with a corresponding transformation intensity according to each first transformation action to obtain the transformed first image data.

[0062] In some implementations, step 102 may be implemented by sequentially transforming the first image data according to the priority of the first transformation operation to obtain the transformed first image data. The priority of the first transformation operation may be set by user or according to the transformation intensity.

[0063] Step 103: input the transformed first image data into an image segmentation model, and output a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located.

[0064] The image segmentation model is used to predict the area where the target object in the image data is located. For example, the image segmentation model can be any basic segmentation model such as SAM, SEEM, etc. The prediction mask can be the outline or boundary of the target object in the first image data. For example, in the application scenario of face recognition, the target object is the face of the person in the image data; in the application scenario of unmanned driving, the target object is the vehicle and road layout in the image data; in the application scenario of industrial product quality inspection, the target object is the product in the image data.

[0065] In some implementations, step 103 may be specifically implemented by: inputting the transformed first image data into an image segmentation model for feature extraction, and outputting a prediction mask of the first image data.

[0066] The performance of basic segmentation models such as SAM and SEEM will drop significantly in different application scenarios. The image data processing method provided by the embodiment of the present disclosure can obtain the optimal transformation operation (the transformation operation that best matches the image data) through the controller regardless of the application scenario of the image data. The image data is transformed by the transformation operation and then input into the image segmentation model for segmentation, which can significantly improve the segmentation accuracy of the image segmentation model. Since the image segmentation model processes the image data after the transformation using the optimal image prompt, the image segmentation model can achieve the expected performance using less image data, reducing the computer resources, data set size and training time required for training. The image data processing method provided by the embodiment of the present disclosure does not require any fine-tuning of the model parameters of the image segmentation model, so that the image segmentation model can be applied to various application scenarios and has a wide range of applications.

[0067] In the disclosed embodiment, for the image data of any application scenario, a plurality of first transformation operations (optimal transformation operations) matching the first image data of the current round are first obtained through the controller, and then the first image data is transformed through the optimal transformation operation, and then the transformed first image data is segmented by the image segmentation model; in this way, no matter which application scenario the image data is, the optimal transformation operation can be obtained through the controller, and the image data is transformed through the transformation operation and then input into the image segmentation model for segmentation, which can significantly improve the segmentation accuracy of the image segmentation model. Since the image segmentation model processes the image data after the transformation using the optimal image prompt, the image segmentation model can achieve the expected performance by using less image data, which reduces the computer resources, data set size and training time required for training. The image data processing method provided by the disclosed embodiment does not require any fine-tuning of the model parameters of the image segmentation model, so that the image segmentation model can be applied to various application scenarios and has a wide range of applications.

[0068] The present disclosure provides an image data processing method, which can be executed by a processor of a computer device. Figure 2 As shown, the method includes the following steps 201 to 205:

[0069] Step 201: For image data of any application scenario, first image data of the current round is mapped to a vector space to obtain a query embedding of the first image data.

[0070] The query embedding of the first image data refers to a vector representation of the first image data.

[0071] In some implementations, a specific implementation of step 201 may be: using an embedding technique to map the first image data to a vector space to obtain a query embedding of the first image data.

[0072] Step 202: embed the query into the multiple prediction heads and output prediction results of the multiple prediction heads; the prediction results are multiple transformation actions matched with the first image data obtained by each prediction head and the transformation strength of each matched transformation action.

[0073] The multiple prediction heads are used to learn image features in the first image data from different angles.

[0074] In some implementations, a specific implementation manner of determining the transformation strength of the transformation action may be: determining the transformation strength of the transformation action based on a property of the transformation action.

[0075] Specifically, when the transformation action is Gaussian blur, the transformation strength of the Gaussian blur is determined based on the Gaussian kernel size and target parameters of the Gaussian blur; the target parameters characterize the degree to which any pixel in the image data is affected by neighboring pixels during Gaussian blur; when the transformation action is linear transformation, the transformation strength of the linear transformation is determined based on the scaling factor and offset of the linear transformation; when the transformation action is brightness transformation, the transformation strength of the brightness transformation is determined based on the adjustment range of the brightness transformation; when the transformation action is saturation transformation, the transformation strength of the saturation transformation is determined based on the adjustment range of the saturation transformation.

[0076] If the transformation action is Gaussian blur, the attributes of the transformation action are the Gaussian kernel size and the target parameter that characterizes the degree to which any pixel in the image data is affected by neighboring pixels during Gaussian blur. For example, if the transformation action is Gaussian blur and the target parameter is a fixed value, the intensity of the Gaussian blur is divided into 0-10 levels according to the Gaussian kernel from small to large. In another example, if the transformation action is Gaussian blur and the target parameter is a variable value, the weight of the Gaussian kernel and the weight of the target parameter are determined, and based on the weight of the Gaussian kernel and the weight of the target parameter, the transformation intensity of the Gaussian blur is divided into 0-10 levels.

[0077] If the transformation action is a linear transformation, the attributes of the transformation action are the scaling factor and offset of the linear transformation. For example, if the transformation action is a linear transformation and the offset is a fixed value, the transformation strength of the transformation action is divided into 0-10 levels according to the scaling factor from small to large. In another example, if the transformation action is a linear transformation and the offset is a variable value, the weight of the linear transformation and the weight of the offset are determined, and based on the weight of the linear transformation and the weight of the offset, the transformation strength of the linear transformation is divided into 0-10 levels.

[0078] If the transformation action is brightness transformation, the attribute of the transformation action is the adjustment range of the brightness transformation. For example, according to the minimum and maximum values ​​of the brightness transformation adjustment, the transformation intensity of the brightness transformation is divided into 0-10 levels.

[0079] If the transformation action is a saturation transformation, the attribute of the transformation action is the adjustment range of the saturation transformation. For example, according to the minimum and maximum values ​​of the saturation adjustment, the transformation intensity of the saturation transformation is divided into 0-10 levels.

[0080] In some implementations, step 202 may be specifically implemented by dividing the query embedding into multiple sub-query embeddings, inputting the multiple sub-query embeddings into multiple prediction heads in sequence, and outputting prediction results of the multiple prediction heads.

[0081] In some embodiments, the specific implementation method of step 202 can also be: input the query embedding into the recurrent neural network of each prediction head respectively, and output the operation embedding corresponding to the query embedding; input the operation embedding corresponding to the query embedding into the feedforward neural network of each prediction head, and output the prediction result of each prediction head.

[0082] The recurrent neural network is used to generate an operation embedding. For example, the recurrent neural network can be LSTM. The feedforward neural network is used to generate the probability of the transformation action and its transformation strength, and can also be used to generate the priority of the transformation action. For example, the feedforward neural network can be a multilayer perceptron (MLP).

[0083] In some embodiments, the specific implementation method of "embedding the query into the corresponding operation embedding input to the feedforward neural network of each prediction head, and outputting the prediction result of each prediction head" can be: embedding the query into the corresponding operation embedding input to the feedforward neural network, and directly outputting the transformation action and transformation strength predicted by each prediction head.

[0084] In some embodiments, the specific implementation method of "embedding the query into the corresponding operation embedding input into the feedforward neural network of each prediction head, and outputting the prediction result of each prediction head" can also be: embedding the query into the corresponding operation embedding input into the feedforward neural network, and outputting the prediction probability of multiple transformation actions determined by each prediction head and the probability of transformation intensity of each transformation action.

[0085] Step 203: Integrate the prediction results of the multiple prediction heads to obtain the first transformation action and the transformation strength of each of the first transformation actions.

[0086] Here, the above steps 201 to 203 correspond to the above step 101, and the specific implementation of the above step 101 may be referred to during implementation.

[0087] In some embodiments, the specific implementation method of step 203 can be: if each prediction head outputs a transformation action and a transformation strength, then the transformation action and transformation strength output by each prediction head are counted, and the m transformation actions with a larger number are taken as the first transformation action, and the transformation strength with the largest number is taken as the first transformation strength.

[0088] In some embodiments, the specific implementation method of step 203 can also be: if each prediction head outputs the predicted probability of multiple transformation actions and the probability of the transformation strength of each transformation action, then the first k preset transformation actions are taken as the first transformation actions, and the transformation strength with the highest probability is taken as the transformation strength of each first transformation action.

[0089] Step 204: transform the first image data based on the first transformation operation to obtain the transformed first image data.

[0090] Here, the above step 204 corresponds to the above step 102, and the specific implementation of the above step 102 may be referred to during implementation.

[0091] Step 205: input the transformed first image data into an image segmentation model, and output a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located.

[0092] Here, the above step 205 corresponds to the above step 103, and the specific implementation of the above step 103 may be referred to during implementation.

[0093] Based on the above embodiments, the image data processing method provided by the embodiment of the present disclosure further includes the following steps 206 to 207:

[0094] Step 206: Adjust the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller; the real mask is the area where the target object is located in the preset first image data.

[0095] The similarity between the predicted mask of the first image data and the real mask of the first image data reflects the influence of the transformation action predicted by the controller on the prediction accuracy of the image segmentation model. Adjusting the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data can keep the controller in an optimal state to obtain the optimal transformation operation.

[0096] In some embodiments, step 206 may be specifically implemented by adjusting the model parameters of the controller based on the intersection-and-union ratio between the predicted mask of the first image data and the real mask of the first image data to obtain an adjusted controller; wherein the intersection-and-union ratio represents the degree of similarity between the predicted mask of the first image data and the real mask of the first image data. The intersection-and-union ratio refers to the value of the intersection of the predicted mask and the real mask divided by the union.

[0097] The intersection of the predicted mask and the true mask refers to the overlapping area between the area where the predicted target object is located and the area where the labeled target object is located. The union of the predicted mask and the true mask refers to the area occupied by the predicted target object area and the labeled target object area as a whole.

[0098] Step 207: input the second image data of the next round into the adjusted controller, output a plurality of second transformation operations matching the second image data, transform the second image data based on the second transformation operations, input the transformed second image data into the image segmentation model, output a predicted mask of the second image data, and continue to adjust the model parameters of the controller based on the similarity between the predicted mask of the second image data and the real mask of the second image data until a preset stop condition is reached.

[0099] The second image data is the image data of the next round. The second transformation operation is a plurality of transformation operations that match the second image data, that is, the second transformation operation may refer to the optimal transformation operation combination sampled. The preset stop condition may be set to the segmentation accuracy of the image segmentation model reaching the accuracy threshold, or may be set to the training round of the model reaching the round threshold, or may be set according to the mask quality and loss value, which is not limited in the embodiments of the present disclosure.

[0100] In some embodiments, the above step 206 may also be implemented through the following steps 2061 to 2062:

[0101] Step 2061: Determine the information entropy of the output probability of the first transformation operation.

[0102] Information entropy is used to measure the uncertainty or randomness of the first transformation operation.

[0103] In some implementations, step 2061 may be specifically implemented as follows: determining a loss value based on the similarity level and the target loss function; determining the information entropy based on the loss value; and the loss value is positively correlated with the information entropy.

[0104] The target loss function is used to measure the difference between the predicted mask of the first image data and the true mask of the first image data.

[0105] In some embodiments, the specific implementation method of "determining the loss value based on the similarity degree and the target loss function" can be: determining the intersection-and-union ratio between the predicted mask of the first image data and the real mask of the first image data; the intersection-and-union ratio represents the similarity degree between the predicted mask of the first image data and the real mask of the first image data; based on the target loss function and the intersection-and-union ratio, determining the loss value.

[0106] In a feasible implementation, the loss value calculation formula can be: ;in, is the enhanced first image data, represents the image segmentation model, m~M represents the real mask, represents the prediction mask; Used to quantify the intersection between the true mask and the predicted mask, that is, the intersection-over-union ratio between the true mask and the predicted mask; represents the target loss function, Represents the loss value.

[0107] In some embodiments, the specific implementation method of "determining the information entropy based on the loss value" can be: when the loss value of the current round is lower than the loss value of the previous round, the information entropy is lowered based on the degree of reduction in the loss value; when the loss value of the current round is higher than the loss value of the previous round, the information entropy is increased based on the degree of increase in the loss value.

[0108] The rationale for determining information entropy based on the loss value is that a lower loss indicates that the transformation operation can predict an accurate mask, which indicates that the information entropy of the output probability of the transformation operation needs to be reduced to reduce the randomness of the transformation operation. Conversely, a higher loss requires increased randomness to explore more transformation combinations, which indicates that the information entropy of the output probability of the transformation operation needs to be increased.

[0109] Step 2062: Adjust the model parameters of the controller based on the information entropy and the similarity to obtain the adjusted controller.

[0110] In some embodiments, the specific implementation method of step 2062 may be: determining the loss value based on the similarity degree and the target loss function; using a gradient descent algorithm to adjust the model parameters of the controller based on the loss value and the information entropy to obtain the adjusted controller.

[0111] In some embodiments, the specific implementation method of "adopting a gradient descent algorithm to adjust the model parameters of the controller based on the loss value and the information entropy to obtain the adjusted controller" can be: adopting a gradient descent algorithm to adjust the model parameters of the controller based on the loss value, information entropy and learning rate to obtain the adjusted controller.

[0112] Example, for model parameters , the following calculation formula can be used to update the parameters : ;in, represents the learning rate, Represents the output probability of the transformation action and the corresponding transformation strength.

[0113] Based on the above embodiments, the image data processing method provided by the embodiment of the present disclosure further includes the following steps 208 to 210:

[0114] Step 208: Determine the mask quality of the first transformation operation based on the similarity between the predicted mask of the first image data and the real mask of the first image data.

[0115] The mask quality refers to the influence of the first transformation operation on the segmentation accuracy of the image segmentation model; the higher the mask quality, the more the first transformation operation can improve the segmentation accuracy of the image segmentation model.

[0116] In some implementations, step 208 may be specifically implemented by determining the mask quality of the first transformation operation based on an intersection-over-union ratio between a predicted mask of the first image data and a real mask of the first image data.

[0117] Specifically, the intersection-and-union ratio between the predicted mask of the first image data and the real mask of the first image data can be directly used as the mask quality of the first transformation operation. Alternatively, the mask quality of the first transformation operation can be determined by the intersection-and-union ratio between the predicted mask of the first image data and the real mask of the first image data, and the corresponding relationship between the intersection-and-union ratio and the mask quality.

[0118] Step 209: When the mask quality of the first transformation operation is higher than the mask quality of the pre-stored target transformation operation, update the target transformation operation to the first transformation operation.

[0119] The target transformation operation is continuously updated and is used to store transformation operations with higher mask quality.

[0120] In a feasible implementation, if the current round is the first round, the target transformation operation is a preset transformation operation; if the current round is not the first round, the target transformation operation may be the transformation operation with the highest mask quality obtained before the current round.

[0121] Step 210: transform the second image data of the next round based on the updated target transformation operation.

[0122] During the training process of the image segmentation model, the mask quality under different transformation operations is recorded, and the transformation operations with higher mask quality are retained. For the reasoning of the validation set or actual application scenarios, the image data is transformed using the transformation operations with higher retained mask quality and input into the image segmentation model to obtain the segmentation result.

[0123] In the disclosed embodiment, for the image data of any application scenario, a plurality of first transformation operations (optimal transformation operations) matching the first image data of the current round are first obtained through the controller, and then the first image data is transformed through the optimal transformation operation, and then the transformed first image data is segmented by the image segmentation model; in this way, no matter which application scenario the image data is, the optimal transformation operation can be obtained through the controller, and the image data is transformed through the transformation operation and then input into the image segmentation model for segmentation, which can significantly improve the segmentation accuracy of the image segmentation model. Since the image segmentation model processes the image data after the transformation using the optimal image prompt, the image segmentation model can achieve the expected performance by using less image data, which reduces the computer resources, data set size and training time required for training. The image data processing method provided by the disclosed embodiment does not require any fine-tuning of the model parameters, so that the image segmentation model can be applied to various application scenarios and has a wide range of applications.

[0124] The following describes the application of the image data processing method provided by the embodiment of the present disclosure in a practical scenario.

[0125] When basic segmentation models such as SAM and SEEM are faced with application scenarios that are different from the distribution of natural images, such as medical images, industrial images, and remote sensing images, the performance of basic segmentation models will drop significantly. These scenarios bring unique challenges, including complex background structures, changing lighting conditions, and object occlusion, which require the adaptability and robustness of the model beyond the capabilities of the basic segmentation model in its original form. Therefore, how to effectively adjust the basic segmentation model to adapt to the challenges brought by these scenarios has become a key research goal.

[0126] The present disclosure embodiment proposes a basic segmentation model adjustment method based on image prompts (the above-mentioned image data processing method), such as Figure 3 As shown, a controller samples a series of transformation strategies (the above transformation operations), and the input image is sent to the basic segmentation model for inference after the combined transformation. The average intersection-over-union ratio of the inferred predicted mask and the true mask is used as a reward to train the controller. The training controller samples the optimal transformation strategy and applies it to the input image to improve the quality of model prediction.

[0127] like Figure 4 As shown, the processing flow of the embodiment of the present disclosure specifically includes:

[0128] a) Initialize the controller, the basic structure of which is LSTM. Each prediction head of the controller is a linear layer, which can predict the probability of the transformation action and transformation intensity corresponding to the image.

[0129] b) Obtain image prompts through the controller. Image prompts consist of a set of transformation strategies (transformation operations), and the transformation strategy consists of specific transformation actions and corresponding transformation strengths. Transformation strategies include Gaussian blur, linear transformation, brightness, and saturation transformation, and the transformation strength is graded from 0 to 10.

[0130] c) Apply the sampled image hint to the input image. For the input image, perform the corresponding strength actions of the transformation strategy in sequence to obtain the transformed image.

[0131] d) Input the transformed image into the basic segmentation model to obtain the model's predicted mask, and use the average intersection-over-union ratio between the predicted mask and the true mask as a reward to train the controller. Basic segmentation models include the common SAM and SEEM.

[0132] e) During the model training process, the mask quality under different image prompts is recorded, and the image prompts with higher mask quality are retained.

[0133] f) For the reasoning of the validation set or actual scene, the image is transformed by retaining the image hint with a higher average intersection-over-union ratio value and input into the image segmentation model to obtain the segmentation result. The transformation strategies include Gaussian blur, linear transformation, brightness, and saturation transformation, and the transformation intensity is graded from 0 to 10 levels.

[0134] The disclosed embodiment proposes a basic segmentation model adjustment method based on image prompts. By searching for the optimal image prompts, it can significantly reduce the computer resources, data and scale required for training, as well as the training time requirements, and effectively improve the performance of the basic segmentation model in different data sets.

[0135] The innovation of the disclosed embodiment is that, through the reinforcement learning method, a combination of image processing methods that effectively prompt the basic segmentation model is searched to find the best prompt (optimal transformation operation), thereby improving the segmentation accuracy of the model on the corresponding data set. The image prompt-based method can adapt the basic segmentation model to various image fields without any fine-tuning of the model parameters.

[0136] The innovative features of the embodiments of the present disclosure can at least achieve the following technical effects:

[0137] Compared with other segmentation model fine-tuning solutions, the method of the disclosed embodiment is efficient and elegant, significantly reducing the computer resources, data set size and training time required for training, and can effectively improve the performance of the basic segmentation model on different data sets.

[0138] Specifically, for the segmentation everything model SAM, the disclosed embodiments improve the average intersection-over-union ratio of 1.99, 0.96 and 1.75 on the camouflage target detection datasets CAMO, COD10K and NC4K, respectively; improve the average intersection-over-union ratio of 0.27 and 9.81 on the industrial defect detection datasets MTSD and KolektorSDD2, respectively; improve the average intersection-over-union ratio of 7.29 and 4.82 on the medical imaging datasets BUSI and Kavisr, respectively; and improve the average intersection-over-union ratio of 5.20 and 4.81 on the heat map related dataset VT1K and the depth map related dataset NJU2K, respectively. For the one-time segmentation of all content models SEEM, the disclosed embodiment improves the average intersection-over-union ratio by 3.09, 0.21 and 0.45 on the camouflaged target detection datasets CAMO, COD10K and NC4K respectively; improves the average intersection-over-union ratio by 12.29 and 28.51 on the industrial defect detection datasets MTSD and KolektorSDD2 respectively; improves the average intersection-over-union ratio by 9.19 and 11.53 on the medical imaging datasets BUSI and Kavisr respectively; and improves the average intersection-over-union ratio by 1.76 and 2.55 on the heat map related dataset VT1K and the depth map related dataset NJU2K respectively.

[0139] Based on the foregoing embodiments, the embodiments of the present disclosure provide an image data processing device, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0140] Figure 5 A schematic diagram of the structure of an image data processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the image data processing device 500 includes: a matching module 510, a transformation module 520 and a prediction module 530, wherein:

[0141] The matching module 510 is configured to input the first image data of the current round into the controller for image data of any application scenario, and output a plurality of first transformation operations matching the first image data; the plurality of first transformation operations include a plurality of identical or different transformation operations; the plurality of identical transformation operations include a plurality of identical transformation actions, but the plurality of transformation factors corresponding to each of the transformation actions are different;

[0142] a transformation module 520, configured to transform the first image data based on the first transformation operation to obtain the transformed first image data;

[0143] The prediction module 530 is configured to input the transformed first image data into an image segmentation model and output a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located.

[0144] In some embodiments, the matching module 510 is specifically configured to: input the first image data into multiple prediction heads in the controller, output multiple first transformation actions matching the first image data and the transformation strength of each of the first transformation actions; the first transformation operation includes the first transformation action and the transformation strength of each of the first transformation actions.

[0145] In some embodiments, the matching module 510 is specifically configured to: input the first image data into multiple prediction heads in the controller, output multiple first transformation actions matching the first image data and the transformation strength of each of the first transformation actions; the first transformation operation includes the first transformation action and the transformation strength of each of the first transformation actions.

[0146] In some embodiments, the matching module 510 is specifically configured to: map the first image data to a vector space to obtain a query embedding of the first image data; input the query embedding to the multiple prediction heads and output prediction results of the multiple prediction heads; the prediction results are multiple transformation actions obtained by each prediction head that match the first image data and the transformation strength of each of the matched transformation actions; integrate the prediction results of the multiple prediction heads to obtain the first transformation action and the transformation strength of each of the first transformation actions.

[0147] In some embodiments, the matching module 510 is specifically configured to: input the query embedding into the recurrent neural network of each prediction head respectively, and output the operation embedding corresponding to the query embedding; input the operation embedding corresponding to the query embedding into the feedforward neural network of each prediction head, and output the prediction result of each prediction head.

[0148] In some embodiments, the matching module 510 is specifically configured to: determine the transformation strength of the transformation action based on the attribute of the transformation action.

[0149] In some embodiments, the matching module 510 is specifically configured as follows: when the transformation action is Gaussian blur, the transformation strength of the Gaussian blur is determined based on the Gaussian kernel size and target parameters of the Gaussian blur; the target parameters characterize the degree to which any pixel in the image data is affected by neighboring pixels during Gaussian blur; when the transformation action is linear transformation, the transformation strength of the linear transformation is determined based on the scaling factor and offset of the linear transformation; when the transformation action is brightness transformation, the transformation strength of the brightness transformation is determined based on the adjustment range of the brightness transformation; when the transformation action is saturation transformation, the transformation strength of the saturation transformation is determined based on the adjustment range of the saturation transformation.

[0150] In some embodiments, the prediction module 530 is specifically configured to: adjust the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller; the real mask is the area where the target object is located in the preset first image data; input the second image data of the next round into the adjusted controller, output multiple second transformation operations matching the second image data, transform the second image data based on the second transformation operations, and input the transformed second image data into the image segmentation model, output the predicted mask of the second image data, and continue to adjust the model parameters of the controller based on the similarity between the predicted mask of the second image data and the real mask of the second image data until the preset stop condition is reached.

[0151] In some embodiments, the prediction module 530 is specifically configured to: determine the information entropy of the output probability of the first transformation operation; and adjust the model parameters of the controller based on the information entropy and the similarity level to obtain the adjusted controller.

[0152] In some embodiments, the prediction module 530 is specifically configured to: determine a loss value based on the similarity level and the target loss function; determine the information entropy based on the loss value; and the loss value is positively correlated with the information entropy.

[0153] In some embodiments, the prediction module 530 is specifically configured to: determine the loss value based on the similarity level and the target loss function; use a gradient descent algorithm to adjust the model parameters of the controller based on the loss value and the information entropy to obtain the adjusted controller.

[0154] In some embodiments, the prediction module 530 is specifically configured to: determine the intersection-and-union ratio between the predicted mask of the first image data and the true mask of the first image data; the intersection-and-union ratio characterizes the degree of similarity between the predicted mask of the first image data and the true mask of the first image data; and determine the loss value based on the target loss function and the intersection-and-union ratio.

[0155] In some embodiments, the prediction module 530 is specifically configured to: determine the mask quality of the first transformation operation based on the similarity between the predicted mask of the first image data and the real mask of the first image data; when the mask quality of the first transformation operation is higher than the mask quality of the pre-stored target transformation operation, update the target transformation operation to the first transformation operation; and transform the second image data of the next round based on the updated target transformation operation.

[0156] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0157] It should be noted that in the embodiments of the present disclosure, if the above-mentioned image data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0158] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0159] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0160] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0161] The embodiments of the present disclosure provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented in hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0162] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product disclosed in the present invention, please refer to the description of the method embodiment disclosed in the present invention for understanding.

[0163] It should be noted that Figure 6 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, such as Figure 6 As shown, the hardware entity of the computer device 600 includes: a processor 601, a communication interface 602 and a memory 603, wherein:

[0164] Processor 601 generally controls the overall operation of computer device 600 .

[0165] The communication interface 602 enables the computer device to communicate with other terminals or servers through a network.

[0166] The memory 603 is configured to store instructions and applications executable by the processor 601, and can also cache data to be processed or processed by the processor 601 and each module in the computer device 600 (for example, first image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data can be transmitted between the processor 601, the communication interface 602, and the memory 603 through the bus 604.

[0167] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in multiple embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The above-mentioned serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0168] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0169] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0170] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present disclosure may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.

Claims

1. A method for processing image data, characterized in that: The image data processing method comprises: For image data of any application scenario, the first image data of the current round is input to the controller, and a plurality of first transformation operations matching the first image data are output; the plurality of first transformation operations include a plurality of identical or different transformation operations; the plurality of identical transformation operations include a plurality of identical transformation actions, but the plurality of transformation factors corresponding to each of the transformation actions are different; transforming the first image data based on the first transformation operation to obtain the transformed first image data; Inputting the transformed first image data into an image segmentation model, and outputting a prediction mask of the first image data; the prediction mask represents the area where the target object in the first image data is located; Adjusting the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller; the real mask is the area where the target object is located in the preset first image data; The second image data of the next round is input into the adjusted controller, a plurality of second transformation operations matching the second image data are output, the second image data is transformed based on the second transformation operations, and the transformed second image data is input into the image segmentation model, a predicted mask of the second image data is output, and the model parameters of the controller are continuously adjusted based on the similarity between the predicted mask of the second image data and the real mask of the second image data until a preset stop condition is reached.

2. The image data processing method according to claim 1, characterized in that: The step of inputting the first image data of the current round into the controller and outputting a plurality of first transformation operations matching the first image data comprises: The first image data is input into a plurality of prediction heads in the controller, and a plurality of first transformation actions matching the first image data and a transformation strength of each of the first transformation actions are output; the first transformation operation includes the first transformation actions and a transformation strength of each of the first transformation actions.

3. The image data processing method according to claim 2, characterized in that: The step of inputting the first image data to a plurality of prediction heads in the controller and outputting a plurality of first transformation actions matching the first image data and a transformation strength of each of the first transformation actions comprises: Mapping the first image data to a vector space to obtain a query embedding of the first image data; Embed the query into the multiple prediction heads, and output prediction results of the multiple prediction heads; the prediction results are multiple transformation actions matched with the first image data and transformation strength of each of the matched transformation actions acquired by each prediction head; The prediction results of the multiple prediction heads are integrated to obtain the first transformation action and the transformation strength of each of the first transformation actions.

4. The image data processing method according to claim 3, characterized in that: The step of embedding the query into the plurality of prediction heads and outputting prediction results of the plurality of prediction heads comprises: Inputting the query embedding into the recurrent neural network of each prediction head respectively, and outputting the operation embedding corresponding to the query embedding; The operation embedding corresponding to the query embedding is input into the feedforward neural network of each prediction head, and the prediction result of each prediction head is output.

5. The image data processing method according to any one of claims 1 to 4, characterized in that: The image data processing method further includes: Based on the properties of the transform action, a transform strength of the transform action is determined.

6. The image data processing method according to claim 5, wherein determining the transformation intensity of the transformation action based on the attribute of the transformation action comprises: When the transformation action is Gaussian blur, determining the transformation intensity of the Gaussian blur based on the Gaussian kernel size and target parameters of the Gaussian blur; The target parameter represents the degree to which any pixel in the image data is affected by neighboring pixels during Gaussian blur; In the case where the transformation action is a linear transformation, determining a transformation strength of the linear transformation based on a scaling factor and an offset of the linear transformation; When the transformation action is brightness transformation, determining a transformation intensity of the brightness transformation based on an adjustment range of the brightness transformation; When the transformation action is a saturation transformation, a transformation strength of the saturation transformation is determined based on an adjustment range of the saturation transformation.

7. The image data processing method according to claim 1, characterized in that: The adjusting the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller includes: determining information entropy of output probabilities of the first transformation operation; The model parameters of the controller are adjusted based on the information entropy and the similarity level to obtain the adjusted controller.

8. The image data processing method according to claim 7, characterized in that: The determining of the information entropy of the output probability of the first transformation operation comprises: Determining a loss value based on the similarity and the target loss function; The information entropy is determined based on the loss value; the loss value is positively correlated with the information entropy.

9. The image data processing method according to claim 7, characterized in that: The adjusting the model parameters of the controller based on the information entropy and the similarity to obtain the adjusted controller includes: Determining a loss value based on the similarity and the target loss function; A gradient descent algorithm is used to adjust the model parameters of the controller based on the loss value and the information entropy to obtain the adjusted controller.

10. The image data processing method according to claim 8 or 9, characterized in that: The determining of the loss value based on the similarity degree and the target loss function includes: Determine an intersection-and-union ratio between a predicted mask of the first image data and a true mask of the first image data; the intersection-and-union ratio represents a degree of similarity between the predicted mask of the first image data and the true mask of the first image data; The loss value is determined based on the target loss function and the intersection-over-union ratio.

11. The image data processing method according to any one of claims 1 to 4, characterized in that: The image data processing method further includes: determining a mask quality of the first transformation operation based on a degree of similarity between a predicted mask of the first image data and a true mask of the first image data; When the mask quality of the first transformation operation is higher than the mask quality of the pre-stored target transformation operation, updating the target transformation operation to the first transformation operation; The second image data of the next round is transformed based on the updated target transformation operation.

12. An image data processing device, characterized in that: The image data processing device comprises: The matching module is configured to input the first image data of the current round into the controller for image data of any application scenario, and output a plurality of first transformation operations matching the first image data; the plurality of first transformation operations include a plurality of identical or different transformation operations; the plurality of identical transformation operations include a plurality of identical transformation actions, but the plurality of transformation factors corresponding to each of the transformation actions are different; a transformation module, configured to transform the first image data based on the first transformation operation to obtain the transformed first image data; A prediction module, configured to input the transformed first image data into an image segmentation model, and output a prediction mask of the first image data; the prediction mask represents an area where a target object in the first image data is located; The prediction module is further configured to adjust the model parameters of the controller based on the similarity between the predicted mask of the first image data and the real mask of the first image data to obtain the adjusted controller; the real mask is a region where the target object is located in the preset first image data; The prediction module is further configured to input the second image data of the next round into the adjusted controller, output a plurality of second transformation operations matching the second image data, transform the second image data based on the second transformation operations, input the transformed second image data into the image segmentation model, output a predicted mask of the second image data, and continue to adjust the model parameters of the controller based on the similarity between the predicted mask of the second image data and the true mask of the second image data until a preset stop condition is reached.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images