Image processing method, device, computer equipment and medium

Abstract

The embodiments of the present application relate to the field of artificial intelligence technology and disclose an image processing method, apparatus, computer equipment, and medium. The method includes: obtaining an image to be processed that includes a target object, performing image segmentation on the image to be processed, and determining a mask image associated with the target object. Furthermore, feature extraction is performed on the image to be processed, and a first prediction value associated with the target object is determined based on the feature extraction results of the image to be processed; feature extraction is performed on the mask image, and a second prediction value associated with the target object is determined based on the feature extraction results of the mask image; and then, a target prediction value associated with the target object is determined based on the first prediction value and the second prediction value. Image segmentation technology can be combined to increase the accuracy of the target prediction value.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, specifically to the field of artificial intelligence technology, and in particular to an image processing method, an image processing device, a computer device, and a computer storage medium. Background Art

[0002] In today's era, with the rapid development of artificial intelligence technology, it not only affects people's production and life in various application fields, but also promotes the development and progress of the world. Taking the medical field as an example, the incidence of scoliosis has increased year by year in recent years. It not only causes cosmetic deformities and psychological problems for adolescents, but can also lead to poor cardiopulmonary function and intractable pain. It is one of the major health issues facing young people in my country. The spine is the axial skeleton of the human body and the pillar of the body. It has functions such as weight bearing, shock absorption, protection, and movement. Accurate measurement of scoliosis angle is of great significance in assisting doctors in diagnosing patients.

[0003] Currently, spinal scoliosis detection primarily relies on X-rays (i.e., images to be processed). Traditionally, scoliosis angle measurement involves manual measurement of the full-length spinal X-ray using a pencil and a protractor, or using a computer-assisted scoliosis angle measurement tool within a PACS (Picture Archiving and Communication Systems) system. When measuring the scoliosis angle using a protractor or computer-assisted tool, examiners typically rely on clinical experience to identify the upper and lower vertebrae with the greatest inclination. These lines are then extended from the vertebral endplates, perpendicular to the line, and measured with a protractor. The degree of curvature is the scoliosis angle. Consequently, traditional full-length spinal X-ray examinations are limited by local X-ray equipment and the experience of medical personnel. This method fails to eliminate the variability inherent in manual measurement, resulting in poor accuracy. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, and medium, which can be combined with image segmentation technology to increase the accuracy of target prediction values.

[0005] In one aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire an image to be processed including a target object;

[0007] Performing image segmentation on the image to be processed to determine a mask image associated with the target object;

[0008] Performing feature extraction on the image to be processed, and determining a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed;

[0009] performing feature extraction on the mask image, and determining a second prediction value associated with the target object according to a result of the feature extraction of the mask image;

[0010] A target prediction value associated with the target object is determined based on the first prediction value and the second prediction value.

[0011] In one aspect, an embodiment of the present application provides an image processing device, comprising:

[0012] An acquisition module, configured to acquire an image to be processed including a target object;

[0013] a segmentation module, configured to perform image segmentation on the image to be processed and determine a mask image associated with the target object;

[0014] a prediction module, configured to extract features from the image to be processed and determine a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed;

[0015] The prediction module is further configured to perform feature extraction on the mask image and determine a second prediction value associated with the target object according to a result of the feature extraction of the mask image;

[0016] The prediction module is further configured to determine a target prediction value associated with the target object based on the first prediction value and the second prediction value.

[0017] In one aspect, an embodiment of the present application provides another image processing method, the method comprising:

[0018] Acquire an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network;

[0019] Acquire a first sample image including a target object and a target label of the first sample image, where the target label indicates a target tag value associated with the target object;

[0020] Performing image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object;

[0021] updating the network parameters of the segmentation network according to the first sample mask image, and iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network;

[0022] Calling the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object;

[0023] Calling the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object;

[0024] Determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0025] updating the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively training the regression network according to the updated network parameters to obtain a target regression network;

[0026] A target image processing model is obtained through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object.

[0027] In one aspect, an embodiment of the present application provides another image processing device, comprising:

[0028] An acquisition module, configured to acquire an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network;

[0029] The acquisition module is further configured to acquire a first sample image including a target object and a target label of the first sample image, wherein the target label indicates a target tag value associated with the target object;

[0030] a training module, configured to perform image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object;

[0031] The training module is further configured to update the network parameters of the segmentation network according to the first sample mask image, and iteratively train the segmentation network according to the updated network parameters to obtain a target segmentation network;

[0032] The training module is further configured to call the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object;

[0033] The training module is further configured to call the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object;

[0034] The training module is further configured to determine a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0035] The training module is further configured to update the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively train the regression network according to the updated network parameters to obtain a target regression network;

[0036] The training module is also used to obtain a target image processing model through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object.

[0037] Correspondingly, an embodiment of the present application further provides a computer device, which includes an output device, a processor and a storage device; the storage device is used to store program instructions; and the processor is used to call the program instructions and execute the above-mentioned image processing method.

[0038] Correspondingly, an embodiment of the present application further provides a computer storage medium, in which program instructions are stored. When the program instructions are executed, they are used to implement the above-mentioned image processing method.

[0039] Accordingly, according to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided above.

[0040] In an embodiment of the present application, a computer device may obtain an image to be processed that includes a target object, perform image segmentation on the image to be processed, and determine a mask image associated with the target object. Furthermore, the computer device may perform feature extraction on the image to be processed and determine a first prediction value associated with the target object based on the feature extraction results of the image to be processed. Feature extraction may also be performed on the mask image and determine a second prediction value associated with the target object based on the feature extraction results of the mask image. Furthermore, a target prediction value associated with the target object may be determined based on the first and second prediction values. Image segmentation techniques may be combined to increase the accuracy of the target prediction value. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 This is a structural diagram of an image processing model provided in an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0044] Figure 3 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of a mask image provided in an embodiment of the present application;

[0046] Figure 5a This is a schematic diagram of the structure of a segmented network provided in an embodiment of the present application;

[0047] Figure 5b This is a schematic diagram of the structure of a regression network provided in an embodiment of the present application;

[0048] Figure 5c This is a schematic structural diagram of a pyramid sampling module provided in an embodiment of the present application;

[0049] Figure 5d This is a schematic structural diagram of another pyramid sampling module provided in an embodiment of the present application;

[0050] Figure 6 This is a flow chart of a joint training process for a segmentation network and a regression network provided by an embodiment of the present application;

[0051] Figure 7 is a flowchart of another image processing method provided in an embodiment of the present application;

[0052] Figure 8 This is a comparison chart of experimental results provided in the examples of this application;

[0053] Figure 9 This is a segmentation result comparison diagram provided in an embodiment of the present application;

[0054] Figure 10 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0055] Figure 11 is a structural diagram of another image processing device provided in an embodiment of the present application;

[0056] Figure 12 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0059] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0060] ML (Machine Learning) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0061] The solutions provided in the embodiments of this application involve machine learning technology of artificial intelligence. Specific examples are provided below to illustrate:

[0062] The embodiment of the present application constructs an image processing model, such as Figure 1As shown, the image processing model includes a segmentation network and a regression network. The segmentation network is used to perform image segmentation on an input image including a target object and determine a mask image associated with the target object; the regression network can be a twin neural network, which has two inputs (the input image and the mask image corresponding to the input image). The two inputs enter two neural networks (a first branch network and a second branch network) respectively. The first branch network extracts features from the input image and determines a first prediction value associated with the target object based on the feature extraction results of the input image; the second branch network extracts features from the mask image and determines a second prediction value associated with the target object based on the feature extraction results of the mask image; and further, a target prediction value associated with the target object is determined based on the first prediction value and the second prediction value.

[0063] After the image processing model is constructed, the image processing model can be trained based on the target task associated with the target object. Subsequently, the trained image processing model (hereinafter referred to as the target image processing model) can be directly used to analyze the image to be processed that includes the target object to determine the target prediction value associated with the target object. In the embodiment of the present application, the segmentation network in the target image processing model can be collectively referred to as the target segmentation network, and the regression network in the target image processing model can be collectively referred to as the target regression network.

[0064] Among them, the specific method of training the image processing model is: obtaining a large number of sample images including the target object and the target label of each sample image, using these sample images and corresponding target labels as training sets, and training the image processing model through the training set to obtain the target image processing model.

[0065] It is understandable that the above-mentioned target image processing model can be applied to any prediction scenario that requires association with a target object, such as the medical field, the biological field, and so on. Taking the medical field as an example, assuming that the prediction scenario is a scoliosis angle prediction scenario, the target task for training the above-mentioned image processing model is: predicting the scoliosis angle of the spine in a spinal scan image (hereinafter collectively referred to as the predicted scoliosis angle). In this case, the above-mentioned target object is the spine, and the spinal scan image is the sample image. The target label added to the sample image includes two parts of information: first, the labeled scoliosis angle; second, mask label information, which indicates the label category of each pixel in the labeled mask image (or it can be understood as the actual mask image) corresponding to the sample image. The label category of each pixel in the labeled mask image can include background, vertebrae, and intervertebral disc. Specifically, each label category can be represented by a different label value. For example, the label values ​​corresponding to pixels of the background, vertebrae, and intervertebral disc categories can be 0, 1, and 2, respectively. This label value can be used to distinguish the categories to which different pixels belong.

[0066] Still taking the medical field as an example, the above prediction scenario can also be a lesion classification prediction scenario (such as thyroid lesion classification, breast lesion classification). Taking the thyroid lesion classification prediction scenario as an example, the target task for training the above image processing model is: accurately predict the thyroid lesion classification in the thyroid image (such as thyroid color ultrasound image). Then, in this case, the above target object is the thyroid gland, and the thyroid color ultrasound image is the sample image. The target label added to the sample image includes two parts of information: first, the lesion area; second, the marked lesion classification corresponding to the lesion area (such as thyroid nodule, thyroid tumor, thyroid cancer, etc.).

[0067] As can be seen from the above content, in the embodiment of the present application, target image processing models applied to different prediction scenarios can be obtained by training different types of sample images. In one embodiment, the computer device can call target image processing models applied to different prediction scenarios, that is, the target image processing models can include multiple ones. In this case, after the computer device obtains the image to be processed, it can first identify the image type of the image to be processed, and select a target image processing model that matches the image type from multiple target image processing models, and then perform data analysis on the above-mentioned image to be processed through the target image processing model that matches the image type to determine the target prediction value associated with the target object (such as scoliosis angle, lesion classification result, etc.).

[0068] For example, taking the case where the target image processing model includes a first image processing model and a second image processing model, the first image processing model is used to determine the scoliosis angle of the spine in a spinal scan image; the second image processing model is used to determine the thyroid lesion area in a thyroid ultrasound image, as well as the lesion classification corresponding to the thyroid lesion area. The image types and output results of each image processing model corresponding to the image to be processed are shown in Table 1. In this case, after the computer device acquires an image to be processed P1, if the image type of the image to be processed P1 is identified as a spinal scan image, the first image processing model can be called to determine the scoliosis angle of the spine in the spinal scan image; if the image type of the image to be processed P1 is identified as a thyroid ultrasound image, the second image processing model can be called to segment the thyroid lesion area from the brain scan image and determine the lesion classification corresponding to the thyroid lesion area.

[0069] Table 1

[0070]

[0071]

[0072] Alternatively, in another embodiment, the computer device runs an image processing platform, such as an application or a webpage. A user can log in to the image processing platform, upload an image to be processed that includes a target object, and enter processing requirement information for the image to be processed. The processing requirement information is used to indicate a target prediction item for the image to be processed. The prediction item may include scoliosis angle, lesion classification, etc., where the symptom classification may be further subdivided into multiple subclassifications, such as thyroid lesion classification, breast lesion classification, etc. Further, the computer device can obtain the image to be processed and the processing requirement information uploaded by the user, select a target image processing model that matches the processing requirement information from multiple target image processing models, and perform data analysis on the image to be processed using the target image processing model that matches the processing requirement information to determine a target prediction value associated with the target object.

[0073] For example, it is assumed that the image processing model includes a first image processing model and a second image processing model. The first image processing model is used to determine the scoliosis angle of the spine in the spinal scan image; the second image processing model is used to determine the thyroid lesion area in the thyroid ultrasound image and the lesion classification corresponding to the thyroid lesion area. The computer device can display the following information: Figure 2 The image processing page to be processed shown in the middle left picture includes multiple prediction items for users to select. Figure 2It can be seen that the user uploaded the spinal scan image and selected the option of scoliosis angle (which can be understood as the user inputting processing requirement information, which indicates that the target prediction item for image 1 is: scoliosis angle). In this case, when the computer device detects that the user has started the processing operation for the spinal scan image (for example, clicking Figure 2 ), the computer device can determine the spinal scan image as the image to be processed, and select a first image processing model from multiple target image processing models as the target image processing model that matches the processing requirement information, and call the first image processing model to determine the scoliosis angle of the spine in the spinal scan image, which scoliosis angle can include the upper thoracic scoliosis angle, the main thoracic scoliosis and the thoracolumbar scoliosis angle.

[0074] Based on the model structure of the target image processing model, the embodiment of the present application proposes a method such as Figure 3 The image processing method shown in FIG. 1 can be executed by a computer device, and the computer device can call the above-mentioned Figure 1 The target image processing model shown in FIG, wherein the computer device may include but is not limited to: tablet computers, laptop computers, notebook computers, desktop computers, etc. Figure 3 As shown, the image processing method may include the following steps S301-S304:

[0075] S301: Acquire an image to be processed including a target object.

[0076] S302: Perform image segmentation on the image to be processed and determine a mask image associated with the target object. In one embodiment, the computer device inputs the image to be processed into the target image processing model and invokes the target segmentation network within the target image processing model to perform image segmentation on the image to obtain a mask image associated with the target object. The mask image is an image that has the same size as the input image to be processed and only retains the region of interest. For example, assuming the target object is the spine, the region of interest here is the spinal region.

[0077] In a specific implementation, when the target segmentation network performs image segmentation on the image to be processed, it can separate parts of the image to be processed with different semantic features, and generate a mask image associated with the target object based on the segmentation result. Taking the image to be processed as a spinal scan image and the target object as the spine as an example, the image to be processed can separate the background, vertebrae and intervertebral disc, and generate a mask image that distinguishes the background area, vertebrae area and intervertebral disc area. Specifically, the category of each pixel point in the mask image may include background, vertebrae or intervertebral disc, and the pixel values ​​corresponding to the pixel points of the categories of background, vertebrae and intervertebral disc may be 0, 1 and 2 respectively, and the pixel value can be used to distinguish the categories to which different pixel points belong.

[0078] For example, the mask image corresponding to the spine scan image can be as follows: Figure 4 As shown in the figure, the background area in the mask image is black, the spinal bone area is white, and the intervertebral disc area is gray. Figure 4 It can be seen that the mask image corresponding to the spine scan image only focuses on the spine area (including the spinal bone area and the intervertebral disc area).

[0079] S303: Perform feature extraction on the image to be processed, and determine a first prediction value associated with the target object according to the feature extraction result of the image to be processed.

[0080] S304: Perform feature extraction on the mask image, and determine a second prediction value associated with the target object according to the feature extraction result of the mask image.

[0081] In one embodiment, the above-mentioned S303 and S304 can be implemented by calling the target regression network in the target image processing model. Specifically, the target regression network can be a twin neural network, which includes a first branch network and a second branch network. The computer device can call the first branch network in the target regression network to perform feature extraction on the above-mentioned image to be processed, and determine the first prediction value associated with the target object based on the feature extraction results of the image to be processed. In addition, the computer device can call the second branch network in the target regression network to perform feature extraction on the mask image obtained in step S302, and determine the second prediction value associated with the target object based on the feature extraction results of the mask image.

[0082] S305: Determine a target prediction value associated with the target object according to the first prediction value and the second prediction value.

[0083] In one embodiment, the first predicted value and the second predicted value may be averaged, and the average of the first predicted value and the second predicted value may be determined as the target predicted value associated with the target object.

[0084] From the above content, it can be seen that the mask image focuses on the region of interest associated with the target object. In the embodiment of the present application, a first prediction value can be determined according to the mask image, a second prediction value can be determined according to the image to be processed, and a target prediction value associated with the target object can be determined in combination with the first prediction value and the second prediction value. In this way, on the one hand, compared with the method of directly obtaining the target prediction value through the image to be processed, more attention can be paid to the region of interest associated with the target object, thereby improving the accuracy of the prediction; on the other hand, compared with the method of directly determining the target prediction value through the mask image, the prediction result of the mask image (i.e., the second prediction value) can be optimized in combination with the prediction result determined according to the image to be processed (i.e., the first prediction value), thereby reducing the influence of the large error of the mask image (e.g., the large deviation between the region of interest in the mask image and the actual region of interest) on the accuracy of the final prediction result.

[0085] In a specific implementation, the target image processing model is based on the target task associated with the target object. Figure 1 The image processing model is obtained by training (as shown), and the image processing model includes a segmentation network and a regression network. When training the image processing model, the segmentation network and the regression network can be trained independently or jointly.

[0086] right Figure 1 The image processing model shown in FIG is further refined. The segmentation network in the above image processing model may include a feature extraction module, a pyramid sampling module and an upsampling module. The model structure of the segmentation network may be as follows: Figure 5a As shown, the feature extraction module can be, for example, a convolutional neural network (CNN), which is used to extract image features of the input image and obtain a feature map; a pyramid sampling module, which is used to extract features from the feature map and obtain a feature map set; an upsampling module, which is used to upsample the feature map set, restore each feature map in the feature map set to the same size as the input image, and determine the mask image corresponding to the input image according to the upsampling result. The first branch network and the second branch network included in the regression network in the above image processing model both include a feature extraction module, a classification activation mapping (CAM) module and a fully connected layer. Exemplarily, the model structure of the regression network can be as follows Figure 5b As shown, the feature extraction modules in the first branch network and the second branch network can both be res18.

[0087] Among them, the structure of the pyramid sampling module can be as follows Figure 5cAs shown, the input feature maps are pooled to the target size corresponding to each layer through N (N is an integer greater than 1) pooling layers to obtain a feature map set, which includes multiple feature maps. For example, N is 4, and the target sizes corresponding to the first pooling layer, the second pooling layer, the third pooling layer and the fourth pooling layer can be: 1×1, 2×2, 3×3 and 6×6, respectively.

[0088] In the semantic segmentation task, we want to extract features from images with a larger receptive field, and at the same time, we want to minimize the loss of resolution of the feature map (too much loss of resolution will result in loss of detailed information about the image boundary). However, these two are contradictory. To obtain a larger receptive field, we need to use a larger convolution kernel or a larger stride when pooling. The former is too computationally intensive, and the latter will result in loss of resolution. Therefore, when the pyramid sampling module adopts Figure 5c In the structure shown, in order to obtain a larger receptive field during feature extraction, a larger step size is usually used during pooling, which results in a lower resolution of the feature map obtained by pooling, affecting the subsequent output results.

[0089] Based on this, we can Figure 5c The pyramid sampling module shown is optimized to obtain Figure 5d The pyramid sampling module shown includes N layers of parallel dilated convolution layers, each of which corresponds to a different dilated convolution rate. For example, if N is 3, the dilated convolution rates corresponding to the first, second, and third dilated convolution layers can be 6, 12, and 18, respectively. In a specific implementation, the pyramid adopts a module that can convolve the input feature map according to the corresponding dilated convolution rate through each dilated convolution layer to obtain a set of feature maps. In this way, by using dilated convolution layers with different dilated convolution rates in parallel, more feature information of the input feature map can be captured, which not only obtains a larger receptive field but also ensures that the resolution of the final feature map is not greatly lost.

[0090] In one embodiment, it is assumed that the segmentation network and the regression network are respectively Figure 5a and Figure 5b As shown in the figure, the target object is the spine, and the target task associated with the target object is to predict the scoliosis angle of the spine in the spinal scan image. In this case, the training process of independently training the segmentation network and the regression network includes the following steps:

[0091] S10. Obtain a training set. Specifically, on the one hand, spinal scan images can be collected and uniformly resized to a specified size (e.g., [512, 256]). The spinal scan images resized to the specified size are then determined as sample images in the training set. Furthermore, the training set can be expanded by randomly flipping, rotating (-45°, 45°), and rescaling the sample images by a factor between (0.85, 1.25). On the other hand, a target label can be determined for each sample image in the training set. This target label can be added after the sample image is determined, or it can be obtained when the spinal scan image is acquired. The target label carries two parts of information: first, the marked scoliosis angle; second, the mask marking information, which indicates the label category of each pixel point in the marked mask image corresponding to the sample image (or can be understood as the actual mask image). The label category of each pixel point in the marked mask image may include background, spine and intervertebral disc. Specifically, each label category can be represented by a different label value. For example, the label values ​​corresponding to the pixel points of the categories of background, spine and intervertebral disc can be 0, 1 and 2 respectively. The label value can be used to distinguish the categories to which different pixel points belong.

[0092] S11. Train the segmentation network using the training set to obtain a trained target segmentation network.

[0093] S12: re-input each sample image in the training set into the trained target segmentation network to determine the mask image corresponding to each sample image.

[0094] S13. Train the regression network according to each sample image and the mask image corresponding to each sample image to obtain a trained target regression network, thereby completing the independent training of the segmentation network and the regression network to obtain a trained target image processing model.

[0095] In another embodiment, assuming that the segmentation network and the regression network are still Figure 5a and Figure 5b As shown, the target object is the spine, and the target task associated with the target object is to predict the scoliosis angle of the spine in the spine scan image. In this case, the training process of jointly training the segmentation network and the regression network (see Figure 6 ), including the following process:

[0096] S20, obtaining a training set. The specific method of obtaining the training set here can refer to the relevant description of the above step S10, which will not be repeated here.

[0097] S21. Obtain a first sample image including a target object from a training set, and obtain a target label for the first sample image, where the target label indicates a target tag value associated with the target object. Here, the first sample image may be a spinal scan image of a specified size, and the target tag value associated with the target object may be a scoliosis angle.

[0098] S22. Perform image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object.

[0099] pass Figure 6 It can be seen that the segmentation network includes a feature extraction module, a pyramid sampling module and an upsampling module. The specific implementation method of step S22 is: extracting the feature map of the first sample image through the feature extraction module in the segmentation network, performing feature extraction on the feature map through the pyramid sampling module to obtain a feature map set, calling the upsampling module to upsample the feature map set, and determining the first sample mask image associated with the target object according to the upsampling result.

[0100] In one embodiment, when the pyramid sampling module is as follows Figure 5c As shown, the input feature maps can be pooled to the target size corresponding to each layer through the pyramid using each pooling layer in the module, thereby obtaining a set of feature maps.

[0101] Alternatively, in another embodiment, when the pyramid sampling module is as follows Figure 5d As shown, each dilated convolution layer in the pyramid sampling module can be used to convolve the feature map according to its corresponding dilated convolution rate to obtain a set of feature maps.

[0102] S23. Perform feature extraction on the first sample image through the first branch network in the regression network, and determine a first sample prediction value associated with the target object according to the feature extraction result of the first sample image.

[0103] In a specific implementation, the feature extraction result of the first sample image can be subjected to classification activation mapping processing to obtain a first classification activation map, and according to the first classification activation map, the first sample prediction value associated with the target object is determined. Among them, the image area associated with the target object is highlighted in the first classification activation map. The first classification activation map here can be understood as a heat map corresponding to the first sample image. The size of the heat map is consistent with the first sample image. The area in the first sample image that has a greater impact on the first sample prediction value (which can be understood as an important area) has a higher heat displayed in the heat map. In an embodiment of the present application, when the output result is the scoliosis angle, the image area with a greater degree of spinal curvature or a more tilted vertebral body is an important area, and the corresponding heat of the important area in the heat map is higher. Among them, when the target object is the spine, the image area associated with the target object highlighted in the first classification activation map is the above-mentioned important area.

[0104] See also Figure 6 As shown, the first branch network includes a first feature extraction module, a first classification activation mapping module, and a first fully connected layer. When executing the above step S23, the image features of the first sample image can be extracted by the first feature extraction module, and the feature extraction results are input into the first classification activation mapping module. The first classification activation mapping module performs classification activation mapping on the feature extraction results to obtain a first classification activation map. Furthermore, the first classification activation map can be subjected to data analysis by the first fully connected layer to determine a first sample prediction value associated with the target object. When the target object is the spine, the first sample prediction value here is the predicted scoliosis angle of the spine in the first sample image.

[0105] S24. Perform feature extraction on the first sample mask image through the second branch network in the regression network, and determine a second sample prediction value associated with the target object according to the feature extraction result of the sample mask image.

[0106] In a specific implementation, the feature extraction result of the first sample mask image can be subjected to classification activation mapping processing to obtain a second classification activation map, and the second sample prediction value associated with the target object can be determined according to the second classification activation map. Among them, the image area associated with the target object is highlighted in the second classification activation map. The second classification activation map here can be understood as a heat map corresponding to the second sample image. The size of the heat map is consistent with that of the first sample mask image. The area in the first sample mask image that has a greater impact on the second sample prediction value (i.e., the important area) has a higher heat displayed in the heat map.

[0107] See also Figure 6As shown, the second branch network includes a second feature extraction module, a second classification activation mapping module, and a second fully connected layer. When executing the above step S24, the image features of the first sample mask image can be extracted by the second feature extraction module, and the feature extraction results can be input into the second classification activation mapping module. The second classification activation mapping module performs classification activation mapping on the feature extraction results to obtain a second classification activation map. Furthermore, the second classification activation map can be subjected to data analysis by the second fully connected layer to determine a second sample prediction value associated with the target object. When the target object is the spine, the second sample prediction value here is the predicted scoliosis angle of the spine in the first sample mask image.

[0108] From the above content, it can be seen that the first classification activation map and the second classification activation map are both derived from the same first sample image. The only difference is that the first classification activation map is obtained directly based on the first sample image, and the second classification activation map is obtained based on the first sample mask image determined by image segmentation of the first sample image. However, in theory, the heat distribution represented by the first classification activation map and the second classification activation map should be consistent, that is, the important areas reflected by the first classification activation map and the second classification activation map (for example, the image area with greater spinal curvature or more tilted vertebral body) should be consistent.

[0109] Based on this, in order to ensure the consistency of the classification activation maps obtained by the first branch network and the second branch network, the embodiment of the present application can obtain the mean absolute value loss function after obtaining the above-mentioned first classification activation map and the second classification activation map, calculate the value of the mean absolute value loss function according to the first classification activation map and the second classification activation map, and update the network parameters of the feature extraction modules (i.e., the above-mentioned first feature extraction module and the second feature extraction module) in the first branch network and the second branch network in the direction of reducing the value of the mean absolute value loss function. Similarly, each time a new sample image and a new sample mask image are input into the first branch network and the second branch network respectively, the value of the mean absolute value loss function can be calculated in the same manner as above, and the network parameters of the feature extraction modules (i.e., the above-mentioned first feature extraction module and the second feature extraction module) in the first branch network and the second branch network can be updated in the direction of reducing the value of the mean absolute value loss function, and so on, until the value of the mean absolute value loss function converges, then the feature extraction modules in the first branch network and the second branch network are stopped from being updated according to the mean absolute value loss function.

[0110] Among them, the mean absolute value loss function Can be:

[0111]

[0112] In formula 1.1, C(x) is the classification activation map obtained by the first branch network, such as the first classification activation map mentioned above, and C(f(x)) is the classification activation map obtained by the second branch network, such as the second classification activation map mentioned above.

[0113] It can be understood that when the value of the mean absolute value loss function reaches convergence, it can be characterized that the classification activation maps obtained by the first branch network and the second branch network are consistent, that is, in this case, the obtained classification activation map can more accurately reflect the actual important areas of the input image (for example, the image area with greater spinal curvature or more inclined vertebrae).

[0114] Based on this, in the embodiment of the present application, during the joint training of the segmentation network and the regression network, as a feasible method, after the value of the mean absolute value loss function reaches convergence, the current classification activation map obtained by the classification activation map module in the first branch network is input into the segmentation network, and the segmentation network is iteratively optimized according to the current classification activation map. The iterative optimization process is as follows:

[0115] Step 1: Obtain a pyramid sampling module to perform feature extraction on the feature map of the input new sample image to obtain a feature extraction result. The new sample image here is the image input into the segmentation network after the sample image corresponding to the current classification activation map.

[0116] Step 2: Get the segmentation network optimization function and calculate the segmentation network optimization function based on the current classification activation map and feature extraction results.

[0117] Step 3: Upsample the calculation result through the upsampling module, and determine the new sample mask image associated with the target object according to the upsampling result. Furthermore, after the segmentation network determines the new sample mask image associated with the target object, the new sample image can be input into the first branch network in the regression network, and the new sample mask image can be input into the second branch network in the regression network. The regression network is trained again with the new sample image and the new sample mask image. In this process, after the first branch network obtains the classification activation map corresponding to the new sample image, the classification activation map corresponding to the new sample image can be input into the segmentation network. The segmentation network performs steps similar to steps S30 to 34 according to the classification activation map corresponding to the new sample image, and continues to iteratively optimize the segmentation network, and so on.

[0118] From the above content, we can see that the classification activation map reflects the important areas of the input image (for example, the image areas with greater spinal curvature or more tilted vertebrae). Training the segmentation network in combination with the classification activation map can make the segmentation network pay more attention to important areas and enhance the segmentation network's segmentation of the region of interest. Accordingly, the mask image output by the segmentation network is also more accurate, and the target prediction value subsequently obtained based on the mask image is also more accurate.

[0119] Step 4: Obtain the mask marking information of the new sample image, and update the network parameters and segmentation network optimization function of the segmentation network according to the new sample mask image and the mask marking information of the new sample image.

[0120] Among them, the segmentation network optimization function is: multiplying the product of the current classification activation map and the feature extraction result by the learning parameter α, and summing the multiplication result and the feature extraction result. The initial value of the learning parameter α is a specified value (for example, 0). The above-mentioned updating of the segmentation network optimization function includes: updating the segmentation network optimization function in the direction of increasing the learning parameter α.

[0121] For example, the segmentation network optimization function f′ is m (x) can be:

[0122] f′ m (x) = α(C(x)*f m (x))+f m (x) Formula 1.2

[0123] In Equation 1.2, C(x) represents the current classification activation map, f m (x) represents the feature extraction result output by the pyramid sampling module. The initial value of the learning parameter α is 0. By gradually increasing it during training, it can be seen from Equation 1.2 that the segmentation network optimization function combines the global view of the input image and selectively aggregates context based on the classification activation map returned by the regression network, thereby improving intra-class compactness and semantic consistency.

[0124] Step 5: Iteratively train the segmentation network according to the updated network parameters to obtain the target segmentation network.

[0125] In the specific implementation, the target loss function of the segmentation network It can be:

[0126]

[0127] Among them, m represents the number of categories of the target to be segmented, f(x j ) and s jThe number of pixels in the i-th category represents the predicted pixel value and the true pixel value, respectively. λ is a weight parameter that can be pre-set based on experimentally measured data. In an embodiment of the present application, when the target object is the spine, in order to make the segmentation network focus on the shape / edge of the spine, each pixel in the mask image output by the segmentation network can be divided into three categories (i.e., m is 3 above): background, vertebrae, and intervertebral discs. The pixel values ​​corresponding to pixels in the background, vertebrae, and intervertebral disc categories can be 0, 1, and 2, respectively, which can be used to distinguish the categories to which different pixels belong.

[0128] In a specific implementation, after the segmentation network obtains a new sample mask image, it can determine the pixel prediction value of each pixel in the new sample mask image, determine the label value (i.e., the above-mentioned true pixel value) of each pixel in the actual mask image corresponding to the new sample image, as indicated by the mask label information corresponding to the new sample image, and calculate the value of the target loss function based on the pixel prediction value and label value. Furthermore, according to the direction of the decrease in the value of the target loss function, the network parameters of the segmentation network and the segmentation network optimization function are updated.

[0129] Alternatively, in the embodiment of the present application, during the joint training of the segmentation network and the regression network, as another feasible approach, after each time a classification activation map is obtained by the first branch network in the regression network, the classification activation map obtained by the first branch network can be input into the segmentation network to iteratively optimize the segmentation network. Specifically, after the first branch network obtains the first classification activation map corresponding to the first sample image, the process of iteratively optimizing the segmentation network is described as follows:

[0130] a. Input the first classification activation map into the segmentation network, and obtain the feature extraction result obtained by the pyramid sampling module on the feature map of the second sample image, where the second sample image is the image input into the segmentation network after the first sample image.

[0131] b. Obtain a segmentation network optimization function and calculate the segmentation network optimization function based on the first classification activation map and feature extraction results.

[0132] c. Upsampling the calculation result through an upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result.

[0133] d. Obtaining the mask marking information of the second sample image, and updating the network parameters and the segmentation network optimization function of the segmentation network according to the second sample mask image and the mask marking information of the second sample image.

[0134] In one embodiment, assuming the target object is a spine, the category of each pixel in the second sample mask image includes background, vertebrae, or intervertebral disc. The second sample mask image distinguishes between the background region, the vertebrae region, and the intervertebral disc region. The mask label information of the second sample image indicates the label category of each pixel in the labeled mask image corresponding to the second sample image, and the label category includes background, vertebrae, or intervertebral disc. The specific implementation of updating the network parameters of the segmentation network according to the mask label information of the second sample mask image and the second sample image can be: calculating the value of the target loss function of the segmentation network according to the mask label information of the second sample mask image and the second sample image, and then updating the network parameters of the segmentation network according to the direction of decrease of the target loss function value. The target loss function can be as shown in Equation 1.3 above. Each pixel in all mask images (including the first mask image, the second mask image, the labeled mask image corresponding to the second sample image, etc.) can be divided into three categories (i.e., m is 3): background, vertebrae, and intervertebral disc. The pixel values ​​corresponding to pixels of the background, vertebrae, and intervertebral disc categories can be 0, 1, and 2, respectively, which can be used to distinguish the categories to which different pixels belong.

[0135] e. Iteratively train the segmentation network based on the updated network parameters to obtain the target segmentation network.

[0136] For the specific implementation of the above a to b, please refer to the above description of steps 1 to 5, which will not be repeated here.

[0137] S25, determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0138] S26. Update the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively train the regression network according to the updated network parameters to obtain the target regression network.

[0139] In one embodiment, the specific implementation of updating the network parameters in step S26 may be: obtaining a regression network loss function, calculating the value of the regression network loss function based on the target sample prediction value and the target label value, and updating the network parameters of the regression network in a direction that reduces the value of the regression network loss function. Furthermore, the regression network may be iteratively trained based on the updated network parameters until the value of the regression network loss function converges, completing the training of the regression network and obtaining a trained target regression network.

[0140] Among them, when the target object is the spine, the target sample prediction value may include any one or more of the following predicted scoliosis angles: predicted upper thoracic scoliosis angle, predicted main thoracic scoliosis angle, and predicted thoracolumbar scoliosis angle; the target label value includes any one or more of the following labeled scoliosis angles: labeled upper thoracic scoliosis angle, labeled main thoracic scoliosis angle, and labeled thoracolumbar scoliosis angle. The above regression network loss function is:

[0141]

[0142] Where i represents the scoliosis angle of category i, which includes upper thoracic scoliosis, main thoracic scoliosis, or thoracolumbar scoliosis. i = 1 represents the category of upper thoracic scoliosis, i = 2 represents the category of main thoracic scoliosis, and i = 3 represents the category of thoracolumbar scoliosis. In this case, n = 3; ∈ is the smoothing factor, y i Characterize the labeled scoliosis angle of category i, g(x i ) represents the predicted scoliosis angle of category i. ∈ is a small value greater than 0, for example, it can be 10 -10 , is to prevent the denominator of the above formula 1.4 from being zero.

[0143] Based on the model structure of the above image processing model, the embodiment of the present application proposes a method such as Figure 7 An image processing method shown in FIG. 1 , which can be executed by a computer device, is shown in FIG. Figure 7 As shown, the image processing method may include the following steps S701-S708:

[0144] S701: Acquire an image processing model, the image processing model includes a segmentation network and a regression network, the regression network includes a first branch network and a second branch network. For example, the model structure of the image processing model can be as follows: Figure 1 shown.

[0145] S702: Acquire a first sample image including a target object and a target label of the first sample image, where the target label indicates a target tag value associated with the target object.

[0146] S703: Perform image segmentation on the first sample image using a segmentation network to determine a first sample mask image associated with the target object. The specific implementation of steps S702 to S703 can be found in the description of steps S21 to S22 in the above embodiment and will not be repeated here.

[0147] S704: updating the network parameters of the segmentation network according to the first sample mask image, and iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network.

[0148] As can be seen from the above, when training the image processing model, the segmentation network and the regression network can be trained independently or jointly. In one embodiment, when the segmentation network and the regression network are trained independently, the specific method of updating the network parameters of the segmentation network according to the first sample mask image includes: obtaining mask labeling information for the first sample image, calculating the value of the target loss function of the segmentation network according to the first sample mask image and the mask labeling information of the first sample image, and further updating the network parameters of the segmentation network according to the direction in which the value of the target loss function decreases.

[0149] In another embodiment, when jointly training a segmentation network and a regression network, the specific method for updating the network parameters of the segmentation network according to the first sample mask image includes: inputting the first classification activation map into the segmentation network, and obtaining a feature extraction result from the feature map of the second sample image by a pyramid sampling module, where the second sample image is the image input into the segmentation network after the first sample image. Furthermore, obtaining a segmentation network optimization function, calculating the segmentation network optimization function based on the first classification activation map and the feature extraction result, upsampling the calculation result through an upsampling module, and determining a second sample mask image associated with the target object based on the upsampling result. Furthermore, obtaining mask tag information of the second sample image, and updating the network parameters of the segmentation network according to the second sample mask image and the mask tag information of the second sample image.

[0150] S705: Calling the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object.

[0151] S706: Call the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object.

[0152] S707: Determine a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value.

[0153] S708: Update the network parameters of the regression network based on the target sample prediction value and the target label value, and iteratively train the regression network based on the updated network parameters to obtain the target regression network. The specific implementation of steps S705 to S708 can be found in the description of steps S23 to S26 in the above embodiment, and will not be repeated here.

[0154] S709: Obtain a target image processing model through a target segmentation network and a target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object.

[0155] In a specific implementation, a target image processing model can be constructed by a target segmentation network and a target regression network. Subsequently, when it is necessary to predict a target prediction value associated with a target object, an image to be processed including the target object can be obtained, and the target segmentation network in the target image processing model can be called to perform image segmentation on the image to be processed, and a mask image associated with the target object can be determined. Furthermore, on the one hand, the first branch network in the target regression network can be called to perform feature extraction on the image to be processed, and the first prediction value associated with the target object can be determined according to the feature extraction result of the image to be processed; on the other hand, the second branch network can be called to perform feature extraction on the mask image, and the second prediction value associated with the target object can be determined according to the feature extraction result of the mask image, and then the target prediction value associated with the target object can be determined based on the first prediction value and the second prediction value. The specific process of joint training can be found in the above-mentioned specific description of joint training, which will not be repeated here.

[0156] Through the above content, the target image processing model proposed in the embodiment of the present application adds a segmentation network, a mean absolute value loss function, and a method for enhancing the region of interest compared to the ordinary image processing model. On the basis of the ordinary image processing model, these methods are superimposed in sequence to conduct a large number of scoliosis angle prediction experiments, and the following can be obtained: Figure 8 The experimental results shown in the figure and Figure 9 The comparison of segmentation results is shown in Figure 2. Figure 8 "Direct regression" represents that the target image processing model only includes the regression network; "segmentation" represents that the segmentation network is added to the target image processing model; "mean absolute value loss function" represents that the above-mentioned mean absolute value loss function is introduced in the process of training the image processing model to obtain the target image processing model; "region of interest enhancement" represents that during the training process, the classification activation map obtained by the first branch network in the regression network is returned to the segmentation network that focuses on the important area (the image area with greater spinal curvature or more tilted vertebrae), thereby increasing the segmentation network's learning of the spinal area and enhancing the accuracy of the segmentation network in segmenting the region of interest (i.e., the spinal area) from the spinal scan image.

[0157] from Figure 8 As shown in the experimental results, it can be seen that the target image processing model proposed in the embodiment of the present application greatly improves the accuracy of predicting scoliosis angle by introducing the segmentation network, the mean absolute value loss function and the method of enhancing the region of interest. Figure 9 As can be seen from the segmentation results shown, the accuracy of the segmentation results output by the segmentation network (i.e., the mask image corresponding to the spine scan image) is greatly increased by adding the region of interest method.

[0158] The following describes the specific application of the image processing method by taking the target application scenario of applying the above-mentioned image processing method to predict the scoliosis angle in the spinal X-ray scan image as an example.

[0159] In the target application scenario, the target object is the spine, and the target prediction value associated with the target object is the predicted scoliosis angle. Specifically, the target image processing model is Figure 1 The image processing model shown is trained, and the target image processing model includes a target segmentation network and a target regression network. The computer device can call the target segmentation network in the target image processing model to perform image segmentation on the spinal X-ray scan image and determine a mask image of the spinal region of interest. The categories of each pixel in the mask image are divided into background, vertebrae, and intervertebral discs. Furthermore, the computer device can use the above-mentioned spinal X-ray scan image and mask image as inputs to the first branch network and the second branch network in the target regression network, respectively. The first branch network performs feature extraction on the spinal X-ray scan image and determines a first predicted scoliosis angle (i.e., the above-mentioned first predicted value) based on the feature extraction results of the spinal X-ray scan image; the second branch network performs feature extraction on the above-mentioned mask image and determines a second predicted scoliosis angle (i.e., the above-mentioned second predicted value) based on the feature extraction results of the mask image. Furthermore, the final predicted scoliosis angle (i.e., the above-mentioned target predicted value) is determined based on the first predicted scoliosis angle and the second predicted scoliosis angle. Doctors can subsequently diagnose the patient's condition based on the predicted scoliosis angle, assisting doctors in diagnosing diseases more quickly.

[0160] As can be seen from the above, the mask image focuses on the spinal region. In the embodiment of the present application, a first predicted scoliosis angle can be determined according to the mask image focusing on the spinal region, a second predicted scoliosis angle can be determined according to the spinal X-ray scan image, and the final predicted scoliosis angle can be determined by combining the first predicted scoliosis angle and the second predicted scoliosis angle. In this way, on the one hand, compared with the method of obtaining the final predicted scoliosis angle directly through the spinal X-ray scan image, more attention can be paid to the spinal region in the process of predicting the scoliosis angle, thereby improving the accuracy of the prediction; on the other hand, compared with the method of determining the final predicted scoliosis angle directly through the mask image, the prediction result of the mask image (i.e., the above-mentioned second predicted scoliosis angle) can be optimized by combining the first predicted scoliosis angle determined according to the original image (i.e., the above-mentioned spinal X-ray scan image), thereby reducing the influence of the large error of the mask image (e.g., the large deviation between the spinal region in the mask image and the actual spinal region) on the accuracy of the final prediction result.

[0161] An embodiment of the present application further provides a computer storage medium, in which program instructions are stored. When the program instructions are executed, they are used to implement the corresponding methods described in the above embodiments.

[0162] See also Figure 10 , is a structural diagram of an image processing device in an embodiment of the present application. The image processing device in an embodiment of the present application can be set in the above-mentioned computer device, or it can be a computer program (including program code) running in the computer device.

[0163] In one implementation of the device of the embodiment of the present application, the device includes the following structure.

[0164] An acquisition module 10 is configured to acquire an image to be processed including a target object;

[0165] The segmentation module 11 is used to perform image segmentation on the image to be processed and determine a mask image associated with the target object;

[0166] A prediction module 12 is configured to extract features from the image to be processed and determine a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed;

[0167] The prediction module 12 is further configured to perform feature extraction on the mask image and determine a second prediction value associated with the target object according to a result of the feature extraction of the mask image;

[0168] The prediction module 12 is further configured to determine a target prediction value associated with the target object based on the first prediction value and the second prediction value.

[0169] In one embodiment, the segmentation module 11 is specifically configured to:

[0170] The target segmentation network is called to perform image segmentation on the image to be processed to obtain a mask image associated with the target object.

[0171] In one embodiment, the prediction module 12 is specifically configured to:

[0172] Calling the first branch network in the target regression network to perform feature extraction on the image to be processed;

[0173] The first prediction value associated with the target object is determined according to a feature extraction result of the image to be processed.

[0174] In one embodiment, the prediction module 12 is further configured to:

[0175] Calling the second branch network in the target regression network to perform feature extraction on the mask image;

[0176] A second prediction value associated with the target object is determined according to a feature extraction result of the mask image.

[0177] In one embodiment, the apparatus further includes a training module 13, the training module 13 being configured to:

[0178] Acquire a first sample image including a target object, and acquire a target label of the first sample image, where the target label indicates a target tag value associated with the target object;

[0179] Performing image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object;

[0180] performing feature extraction on the first sample image through a first branch network in a regression network, and determining a first sample prediction value associated with the target object according to a result of the feature extraction of the first sample image;

[0181] performing feature extraction on the first sample mask image through a second branch network in the regression network, and determining a second sample prediction value associated with the target object according to a result of the feature extraction of the sample mask image;

[0182] Determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0183] According to the target sample prediction value and the target label value, the network parameters of the regression network are updated, and the regression network is iteratively trained according to the updated network parameters to obtain a target regression network.

[0184] In one embodiment, the training module 13 is specifically configured to:

[0185] performing classification activation mapping processing on a feature extraction result of the first sample image to obtain a first classification activation map, wherein the first classification activation map highlights an image region associated with the target object;

[0186] A first sample prediction value associated with the target object is determined according to the first classification activation map.

[0187] In one embodiment, the segmentation network includes a feature extraction module, a pyramid sampling module, and an upsampling module. The training module 13 is further specifically configured to:

[0188] Extracting a feature map of the first sample image by a feature extraction module in the segmentation network;

[0189] Performing feature extraction on the feature map by using the pyramid sampling module to obtain a feature map set;

[0190] The upsampling module is called to upsample the feature map set, and a first sample mask image associated with the target object is determined according to the upsampling result.

[0191] In one embodiment, the pyramid sampling module includes multiple layers of parallel dilated convolution layers, each layer of dilated convolution layers corresponds to a different dilated convolution rate, and the training module 13 is further specifically used to: through each layer of dilated convolution layers in the pyramid sampling module, convolve the feature map according to the corresponding dilated convolution rate to obtain a feature map set.

[0192] In one embodiment, the training module 13 is further configured to:

[0193] Inputting the first classification activation map into the segmentation network, and obtaining a feature extraction result obtained by performing feature extraction on a feature map of a second sample image by the pyramid sampling module, where the second sample image is an image input into the segmentation network after the first sample image;

[0194] Obtaining a segmentation network optimization function, and calculating the segmentation network optimization function according to the first classification activation map and the feature extraction result;

[0195] Upsampling the calculation result by the upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result;

[0196] Obtaining mask marking information of the second sample image, and updating network parameters of the segmentation network and the segmentation network optimization function according to the second sample mask image and the mask marking information of the second sample image;

[0197] The segmentation network is iteratively trained according to the updated network parameters to obtain a target segmentation network.

[0198] In one embodiment, both the first branch network and the second branch network include a feature extraction module; the feature extraction module in the first branch network is used to extract features from the first sample image; the feature extraction module in the second branch network is used to extract features from the sample mask image; the second sample prediction value is determined based on a second classification activation map obtained by performing classification activation mapping processing on the feature extraction result of the sample mask image; and the training module 13 is further specifically used to:

[0199] Get the mean absolute value loss function;

[0200] Calculating a value of the mean absolute value loss function based on the first classification activation map and the second classification activation map;

[0201] The network parameters of the feature extraction modules in the first branch network and the second branch network are updated in a direction of reducing the value of the mean absolute value loss function.

[0202] In one embodiment, the segmentation network optimization function is: multiplying the product of the first classification activation map and the feature extraction result by a learning parameter α, and summing the multiplication result and the feature extraction result. The initial value of the learning parameter α is a specified value. The training module 13 is further specifically used to:

[0203] The segmentation network optimization function is updated in the direction of increasing the learning parameter α.

[0204] In one embodiment, the training module 13 is further configured to:

[0205] Get the regression network loss function;

[0206] Calculating the value of the regression network loss function according to the target sample prediction value and the target label value;

[0207] The network parameters of the regression network are updated in a direction of reducing the value of the regression network loss function.

[0208] In one embodiment, the target object is a spine, the target sample prediction value includes any one or more of the following predicted scoliosis angles: predicted upper thoracic scoliosis angle, predicted main thoracic scoliosis angle, and predicted thoracolumbar scoliosis angle; the target label value includes any one or more of the following labeled scoliosis angles: labeled upper thoracic scoliosis angle, labeled main thoracic scoliosis angle, and labeled thoracolumbar scoliosis angle;

[0209] The regression network loss function is:

[0210]

[0211] Where i represents the scoliosis angle of category i, and the scoliosis angle of category i includes upper thoracic scoliosis angle, main thoracic scoliosis angle or thoracolumbar scoliosis angle; ∈ is the smoothing factor, y i Characterize the labeled scoliosis angle of category i, g(x i ) represents the predicted scoliosis angle of category i.

[0212] In one embodiment, the target object is a spine, the category of each pixel in the mask image includes background, spine, or intervertebral disc, the mask image distinguishes between a background area, a spine area, and an intervertebral disc area, the mask labeling information indicates a labeling category of each pixel in the labeled mask image corresponding to the second sample image, and the labeling category includes background, spine, or intervertebral disc; the training module 13 is further specifically configured to:

[0213] Calculating a value of an objective loss function of the segmentation network according to the second sample mask image and the mask marking information of the second sample image;

[0214] The network parameters of the segmentation network are updated according to the direction in which the value of the objective loss function decreases.

[0215] In the embodiments of the present application, the specific implementation of each of the above modules can refer to the description of the relevant content in the embodiments corresponding to the aforementioned figures.

[0216] The image processing device in the embodiments of the present application can obtain an image to be processed that includes a target object, perform image segmentation on the image to be processed, and determine a mask image associated with the target object. Furthermore, the device can perform feature extraction on the image to be processed and determine a first prediction value associated with the target object based on the feature extraction results of the image to be processed. Feature extraction can also be performed on the mask image and determine a second prediction value associated with the target object based on the feature extraction results of the mask image. Furthermore, a target prediction value associated with the target object can be determined based on the first and second prediction values. Image segmentation techniques can be combined to increase the accuracy of the target prediction value.

[0217] See also Figure 11 , is a structural diagram of an image processing device in an embodiment of the present application. The image processing device in an embodiment of the present application can be set in the above-mentioned computer device, or it can be a computer program (including program code) running in the computer device.

[0218] In one implementation of the device of the embodiment of the present application, the device includes the following structure.

[0219] An acquisition module 20 is configured to acquire an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network;

[0220] The acquisition module 20 is further configured to acquire a first sample image including a target object and a target label of the first sample image, wherein the target label indicates a target tag value associated with the target object;

[0221] A training module 21 is configured to perform image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object;

[0222] The training module 21 is further configured to update the network parameters of the segmentation network according to the first sample mask image, and iteratively train the segmentation network according to the updated network parameters to obtain a target segmentation network;

[0223] The training module 21 is further configured to call the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object;

[0224] The training module 21 is further configured to call the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object;

[0225] The training module 21 is further configured to determine a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0226] The training module 21 is further configured to update the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively train the regression network according to the updated network parameters to obtain a target regression network;

[0227] The training module 21 is further used to obtain a target image processing model through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object.

[0228] See also Figure 12 , is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device according to the embodiment of the present application includes a power supply module and other structures, and includes a processor 70, a storage device 71, and an output device 72. The processor 70, the storage device 71, and the output device 72 can exchange data with each other, and the processor 70 implements the corresponding image processing function.

[0229] The storage device 71 may include a volatile memory, such as a random-access memory (RAM); the storage device 71 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 71 may also include a combination of the above types of memory.

[0230] The processor 70 may be a central processing unit (CPU). In one embodiment, the processor 70 may also be a graphics processing unit (GPU). The processor 70 may also be a combination of a CPU and a GPU. In a computer device, multiple CPUs and GPUs may be included as needed to perform corresponding image processing.

[0231] The output device 72 may include a display (LCD, etc.), a speaker, etc., and may be used to output a target prediction value associated with the target object.

[0232] In one embodiment, the storage device 71 is used to store program instructions. The processor 70 can call the program instructions to implement the various methods mentioned above in the embodiments of the present application.

[0233] In a first possible implementation, the processor 70 of the computer device calls program instructions stored in the storage device 71 to obtain an image to be processed including a target object;

[0234] Performing image segmentation on the image to be processed to determine a mask image associated with the target object;

[0235] Performing feature extraction on the image to be processed, and determining a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed;

[0236] performing feature extraction on the mask image, and determining a second prediction value associated with the target object according to a result of the feature extraction of the mask image;

[0237] A target prediction value associated with the target object is determined based on the first prediction value and the second prediction value.

[0238] In one embodiment, the processor 70 is specifically configured to:

[0239] The target segmentation network is called to perform image segmentation on the image to be processed to obtain a mask image associated with the target object.

[0240] In one embodiment, the processor 70 is specifically configured to:

[0241] Calling the first branch network in the target regression network to perform feature extraction on the image to be processed;

[0242] The first prediction value associated with the target object is determined according to a feature extraction result of the image to be processed.

[0243] In one embodiment, the processor 70 is further configured to:

[0244] Calling the second branch network in the target regression network to perform feature extraction on the mask image;

[0245] A second prediction value associated with the target object is determined according to a feature extraction result of the mask image.

[0246] In one embodiment, the processor 70 is further configured to:

[0247] Acquire a first sample image including a target object, and acquire a target label of the first sample image, where the target label indicates a target tag value associated with the target object;

[0248] Performing image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object;

[0249] performing feature extraction on the first sample image through a first branch network in a regression network, and determining a first sample prediction value associated with the target object according to a result of the feature extraction of the first sample image;

[0250] performing feature extraction on the first sample mask image through a second branch network in the regression network, and determining a second sample prediction value associated with the target object according to a result of the feature extraction of the sample mask image;

[0251] Determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value;

[0252] According to the target sample prediction value and the target label value, the network parameters of the regression network are updated, and the regression network is iteratively trained according to the updated network parameters to obtain a target regression network.

[0253] In one embodiment, the processor 70 is specifically configured to:

[0254] performing classification activation mapping processing on a feature extraction result of the first sample image to obtain a first classification activation map, wherein the first classification activation map highlights an image region associated with the target object;

[0255] A first sample prediction value associated with the target object is determined according to the first classification activation map.

[0256] In one embodiment, the segmentation network includes a feature extraction module, a pyramid sampling module, and an upsampling module, and the processor 70 is further specifically configured to:

[0257] Extracting a feature map of the first sample image by a feature extraction module in the segmentation network;

[0258] Performing feature extraction on the feature map by using the pyramid sampling module to obtain a feature map set;

[0259] The upsampling module is called to upsample the feature map set, and a first sample mask image associated with the target object is determined according to the upsampling result.

[0260] In one embodiment, the pyramid sampling module includes multiple layers of parallel dilated convolution layers, each layer of dilated convolution layers corresponds to a different dilated convolution rate, and the processor 70 is further specifically used to: through each layer of dilated convolution layers in the pyramid sampling module, convolve the feature map according to the corresponding dilated convolution rate to obtain a feature map set.

[0261] In one embodiment, the processor 70 is further configured to:

[0262] Inputting the first classification activation map into the segmentation network, and obtaining a feature extraction result obtained by performing feature extraction on a feature map of a second sample image by the pyramid sampling module, where the second sample image is an image input into the segmentation network after the first sample image;

[0263] Obtaining a segmentation network optimization function, and calculating the segmentation network optimization function according to the first classification activation map and the feature extraction result;

[0264] Upsampling the calculation result by the upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result;

[0265] Obtaining mask marking information of the second sample image, and updating network parameters of the segmentation network and the segmentation network optimization function according to the second sample mask image and the mask marking information of the second sample image;

[0266] The segmentation network is iteratively trained according to the updated network parameters to obtain a target segmentation network.

[0267] In one embodiment, both the first branch network and the second branch network include a feature extraction module; the feature extraction module in the first branch network is used to extract features from the first sample image; the feature extraction module in the second branch network is used to extract features from the sample mask image; the second sample prediction value is determined based on a second classification activation map obtained by performing classification activation mapping processing on the feature extraction result of the sample mask image; the processor 70 is further specifically used to:

[0268] Get the mean absolute value loss function;

[0269] Calculating a value of the mean absolute value loss function based on the first classification activation map and the second classification activation map;

[0270] The network parameters of the feature extraction modules in the first branch network and the second branch network are updated in a direction of reducing the value of the mean absolute value loss function.

[0271] In one embodiment, the segmentation network optimization function is: multiplying the product of the first classification activation map and the feature extraction result by a learning parameter α, and summing the multiplication result and the feature extraction result. The initial value of the learning parameter α is a specified value. The processor 70 is further specifically configured to:

[0272] The segmentation network optimization function is updated in the direction of increasing the learning parameter α.

[0273] In one embodiment, the processor 70 is further configured to:

[0274] Get the regression network loss function;

[0275] Calculating the value of the regression network loss function according to the target sample prediction value and the target label value;

[0276] The network parameters of the regression network are updated in a direction of reducing the value of the regression network loss function.

[0277] In one embodiment, the target object is a spine, the target sample prediction value includes any one or more of the following predicted scoliosis angles: predicted upper thoracic scoliosis angle, predicted main thoracic scoliosis angle, and predicted thoracolumbar scoliosis angle; the target label value includes any one or more of the following labeled scoliosis angles: labeled upper thoracic scoliosis angle, labeled main thoracic scoliosis angle, and labeled thoracolumbar scoliosis angle;

[0278] The regression network loss function is:

[0279]

[0280] Where i represents the scoliosis angle of category i, and the scoliosis angle of category i includes upper thoracic scoliosis angle, main thoracic scoliosis angle or thoracolumbar scoliosis angle; ∈ is the smoothing factor, y i Characterize the labeled scoliosis angle of category i, g(x i ) represents the predicted scoliosis angle of category i.

[0281] In one embodiment, the target object is a spine, the category of each pixel in the mask image includes background, vertebrae, or intervertebral disc, the mask image distinguishes between a background region, a vertebrae region, and an intervertebral disc region, the mask labeling information indicates a labeling category of each pixel in the labeled mask image corresponding to the second sample image, the labeling category including background, vertebrae, or intervertebral disc; and the processor 70 is further specifically configured to:

[0282] Calculating a value of an objective loss function of the segmentation network according to the second sample mask image and the mask marking information of the second sample image;

[0283] The network parameters of the segmentation network are updated according to the direction in which the value of the objective loss function decreases.

[0284] In another possible embodiment, the processor 70 of the computer device calls the program instructions stored in the storage device 71 to obtain an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network; obtains a first sample image including a target object and a target label of the first sample image, wherein the target label indicates a target tag value associated with the target object; performs image segmentation on the first sample image through the segmentation network to determine a first sample mask image associated with the target object; updates the network parameters of the segmentation network according to the first sample mask image, and iteratively trains the segmentation network according to the updated network parameters to obtain a target segmentation network; calls the first branch network to perform image segmentation on the first sample image; The method comprises the following steps: performing feature extraction on the first sample mask image to determine a first sample prediction value associated with the target object; calling the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object; determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value; updating the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively training the regression network according to the updated network parameters to obtain a target regression network; obtaining a target image processing model through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object.

[0285] In the embodiment of the present application, the specific implementation of the above-mentioned processor 70 can refer to the description of the relevant content in the embodiments corresponding to the aforementioned figures.

[0286] The computer device in the embodiments of the present application can obtain an image to be processed that includes a target object, perform image segmentation on the image to be processed, and determine a mask image associated with the target object. Furthermore, the computer device can perform feature extraction on the image to be processed and determine a first prediction value associated with the target object based on the feature extraction results of the image to be processed. Feature extraction can also be performed on the mask image and determine a second prediction value associated with the target object based on the feature extraction results of the mask image. Furthermore, a target prediction value associated with the target object can be determined based on the first and second prediction values. Image segmentation techniques can be combined to increase the accuracy of the target prediction value.

[0287] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The described program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0288] The above disclosure is only part of the embodiments of the present application, and it is certainly not intended to limit the scope of the rights of the present application. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the invention.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed including a target object; Invoking a target segmentation network in a target image processing model to perform image segmentation on the image to be processed and determining a mask image associated with the target object; the target segmentation network is obtained by jointly training a segmentation network and a regression network, and iteratively optimizing the segmentation network during the joint training process, the segmentation network including a feature extraction module, a pyramid sampling module, and an upsampling module; Performing feature extraction on the image to be processed, and determining a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed; performing feature extraction on the mask image, and determining a second prediction value associated with the target object according to a result of the feature extraction of the mask image; determining a target prediction value associated with the target object based on the first prediction value and the second prediction value; Iteratively optimizing the segmentation network includes: inputting a first classification activation map into the segmentation network, and obtaining a feature extraction result obtained by performing feature extraction on a feature map of a second sample image by the pyramid sampling module, wherein the first classification activation map is obtained by performing classification activation mapping processing on a feature extraction result of a first sample image containing the target object, and the second sample image is an image input into the segmentation network after the first sample image; obtaining a segmentation network optimization function, and calculating the segmentation network optimization function according to the first classification activation map and the feature extraction result; upsampling the calculation result through the upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result; obtaining mask labeling information of the second sample image, and updating the network parameters of the segmentation network and the segmentation network optimization function according to the second sample mask image and the mask labeling information of the second sample image; iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network.

2. The method according to claim 1, wherein The target image processing model further includes a target regression network, the first prediction value is determined by extracting features of the image to be processed through a first branch network in the target regression network and according to the feature extraction result of the image to be processed; the second prediction value is determined by extracting features of the mask image through a second branch network in the target regression network and according to the feature extraction result of the mask image, and the method further includes: Acquire a first sample image including a target object, and acquire a target label of the first sample image, where the target label indicates a target tag value associated with the target object; Performing image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object; performing feature extraction on the first sample image through a first branch network in a regression network, and determining a first sample prediction value associated with the target object according to a result of the feature extraction of the first sample image; performing feature extraction on the first sample mask image through a second branch network in the regression network, and determining a second sample prediction value associated with the target object according to a result of the feature extraction of the sample mask image; Determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value; According to the target sample prediction value and the target label value, the network parameters of the regression network are updated, and the regression network is iteratively trained according to the updated network parameters to obtain a target regression network.

3. The method according to claim 2, wherein The determining, according to the feature extraction result of the first sample image, a first sample prediction value associated with the target object includes: performing classification activation mapping processing on a feature extraction result of the first sample image to obtain a first classification activation map, wherein the first classification activation map highlights an image region associated with the target object; A first sample prediction value associated with the target object is determined according to the first classification activation map.

4. The method according to claim 3, wherein The performing image segmentation on the first sample image by using a segmentation network to determine a first sample mask image associated with the target object includes: Extracting a feature map of the first sample image by a feature extraction module in the segmentation network; Performing feature extraction on the feature map by using the pyramid sampling module to obtain a feature map set; The upsampling module is called to upsample the feature map set, and a first sample mask image associated with the target object is determined according to the upsampling result.

5. The method according to claim 4, wherein The pyramid sampling module includes multiple layers of parallel hole convolution layers, each hole convolution layer corresponds to a different hole convolution rate, and the feature map is extracted from the feature map by the pyramid sampling module to obtain a feature map set, including: Through each dilated convolution layer in the pyramid sampling module, the feature map is convolved according to the corresponding dilated convolution rate to obtain a feature map set.

6. The method according to claim 2, wherein The first branch network and the second branch network both include a feature extraction module; the feature extraction module in the first branch network is used to extract features from the first sample image; the feature extraction module in the second branch network is used to extract features from the sample mask image; the second sample prediction value is determined based on a second classification activation map obtained by performing classification activation mapping processing on the feature extraction result of the sample mask image. The method further comprises: Get the mean absolute value loss function; Calculating a value of the mean absolute value loss function based on the first classification activation map and the second classification activation map; The network parameters of the feature extraction modules in the first branch network and the second branch network are updated in a direction of reducing the value of the mean absolute value loss function.

7. The method according to claim 1, wherein The segmentation network optimization function is: multiplying the product of the first classification activation map and the feature extraction result by a learning parameter α, and summing the multiplication result and the feature extraction result, wherein the initial value of the learning parameter α is a specified value. Updating the segmentation network optimization function includes: The segmentation network optimization function is updated in the direction of increasing the learning parameter α.

8. The method according to claim 2, wherein The updating of the network parameters of the regression network according to the target sample prediction value and the target label value includes: Get the regression network loss function; Calculating the value of the regression network loss function according to the target sample prediction value and the target label value; The network parameters of the regression network are updated in a direction of reducing the value of the regression network loss function.

9. The method according to claim 1, wherein The target object is a spine, the category of each pixel point in the second sample mask image includes background, spine, or intervertebral disc, the second sample mask image distinguishes between a background area, a spine area, and an intervertebral disc area, and the mask labeling information indicates a labeling category of each pixel point in the labeling mask image corresponding to the second sample image, the labeling category including background, spine, or intervertebral disc; The updating of the network parameters of the segmentation network according to the second sample mask image and the mask marking information of the second sample image includes: Calculating a value of an objective loss function of the segmentation network according to the second sample mask image and the mask marking information of the second sample image; The network parameters of the segmentation network are updated according to the direction in which the value of the objective loss function decreases.

10. An image processing method, characterized in that: include: Acquire an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network; Acquire a first sample image including a target object and a target label of the first sample image, where the target label indicates a target tag value associated with the target object; Performing image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object; updating the network parameters of the segmentation network according to the first sample mask image, and iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network; Calling the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object; Calling the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object; Determining a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value; updating the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively training the regression network according to the updated network parameters to obtain a target regression network; A target image processing model is obtained through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object; the target segmentation network is obtained by jointly training the segmentation network and the regression network, and in the process of the joint training, the segmentation network is iteratively optimized, and the segmentation network includes a feature extraction module, a pyramid sampling module and an upsampling module; iterative optimization of the segmentation network includes: inputting a first classification activation map into the segmentation network, and obtaining the feature extraction result obtained by the pyramid sampling module for the feature map of the second sample image, wherein the first classification activation map is an image containing the target object. The method comprises the following steps: performing classification activation mapping processing on a feature extraction result of a first sample image of the target object, wherein the second sample image is an image input into the segmentation network after the first sample image; obtaining a segmentation network optimization function, and calculating the segmentation network optimization function according to the first classification activation map and the feature extraction result; upsampling the calculation result through the upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result; obtaining mask labeling information of the second sample image, and updating the network parameters of the segmentation network and the segmentation network optimization function according to the second sample mask image and the mask labeling information of the second sample image; iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network.

11. An image processing device, characterized in that: include: An acquisition module, configured to acquire an image to be processed including a target object; a segmentation module, configured to call a target segmentation network in a target image processing model to perform image segmentation on the image to be processed and determine a mask image associated with the target object; the target segmentation network is obtained by jointly training a segmentation network and a regression network and iteratively optimizing the segmentation network during the joint training process; the segmentation network includes a feature extraction module, a pyramid sampling module, and an upsampling module; a prediction module, configured to extract features from the image to be processed and determine a first prediction value associated with the target object according to a result of the feature extraction of the image to be processed; The prediction module is further configured to perform feature extraction on the mask image and determine a second prediction value associated with the target object according to a result of the feature extraction of the mask image; The prediction module is further configured to determine a target prediction value associated with the target object based on the first prediction value and the second prediction value; Iteratively optimizing the segmentation network includes: inputting a first classification activation map into the segmentation network, and obtaining a feature extraction result obtained by performing feature extraction on a feature map of a second sample image by the pyramid sampling module, wherein the first classification activation map is obtained by performing classification activation mapping processing on a feature extraction result of a first sample image containing the target object, and the second sample image is an image input into the segmentation network after the first sample image; obtaining a segmentation network optimization function, and calculating the segmentation network optimization function according to the first classification activation map and the feature extraction result; upsampling the calculation result through the upsampling module, and determining a second sample mask image associated with the target object according to the upsampling result; obtaining mask labeling information of the second sample image, and updating the network parameters of the segmentation network and the segmentation network optimization function according to the second sample mask image and the mask labeling information of the second sample image; iteratively training the segmentation network according to the updated network parameters to obtain a target segmentation network.

12. An image processing device, characterized in that: include: An acquisition module, configured to acquire an image processing model, wherein the image processing model includes a segmentation network and a regression network, and the regression network includes a first branch network and a second branch network; The acquisition module is further configured to acquire a first sample image including a target object and a target label of the first sample image, wherein the target label indicates a target tag value associated with the target object; a training module, configured to perform image segmentation on the first sample image through a segmentation network to determine a first sample mask image associated with the target object; The training module is further configured to update the network parameters of the segmentation network according to the first sample mask image, and iteratively train the segmentation network according to the updated network parameters to obtain a target segmentation network; The training module is further configured to call the first branch network to perform feature extraction on the first sample image to determine a first sample prediction value associated with the target object; The training module is further configured to call the second branch network to perform feature extraction on the first sample mask image to determine a second sample prediction value associated with the target object; The training module is further configured to determine a target sample prediction value associated with the target object according to the first sample prediction value and the second sample prediction value; The training module is further configured to update the network parameters of the regression network according to the target sample prediction value and the target label value, and iteratively train the regression network according to the updated network parameters to obtain a target regression network; The training module is further used to obtain a target image processing model through the target segmentation network and the target regression network, wherein the target image processing model is used to perform data analysis on the image to be processed including the target object to obtain a target prediction value associated with the target object; the target segmentation network is obtained by jointly training the segmentation network and the regression network, and in the process of the joint training, the segmentation network is iteratively optimized, the segmentation network includes a feature extraction module, a pyramid sampling module and an upsampling module; iterative optimization of the segmentation network includes: inputting a first classification activation map into the segmentation network, and obtaining the feature extraction result obtained by the pyramid sampling module for the feature map of the second sample image, wherein the first classification activation map is a feature map of the image including The feature extraction result of the first sample image of the target object is subjected to classification activation mapping processing, and the second sample image is an image input into the segmentation network after the first sample image; a segmentation network optimization function is obtained, and the segmentation network optimization function is calculated according to the first classification activation map and the feature extraction result; the calculation result is upsampled by the upsampling module, and a second sample mask image associated with the target object is determined according to the upsampling result; mask labeling information of the second sample image is obtained, and the network parameters of the segmentation network and the segmentation network optimization function are updated according to the second sample mask image and the mask labeling information of the second sample image; the segmentation network is iteratively trained according to the updated network parameters to obtain a target segmentation network.

13. A computer device, characterized in that: The computer device includes a processor and a storage device, which are connected to each other, wherein the storage device is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 10.

14. A computer storage medium, characterized in that The computer storage medium stores program instructions, which, when executed, are used to implement the method according to any one of claims 1 to 10.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer storage medium. A processor of a computer device reads the computer instructions from the computer storage medium and executes the computer instructions, so that the computer device performs the method according to any one of claims 1 to 10.

Images