Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method, an image processing device, electronic equipment, a computer storage medium and a computer program product, and the method comprises the steps: carrying out the initial edge feature extraction processing of any scanning image in a scanning image sequence after obtaining the scanning image sequence, and obtaining an initial edge feature graph; predicting a reference edge feature map of any scanned image based on adjacent images of any scanned image and the forward change relation; predicting an adjacent edge feature map of an adjacent image of the any scanned image based on the any scanned image and the reverse change relation; and finally, according to the initial edge feature map, the reference edge feature map and the adjacent edge feature map, determining a target image area presenting the section of the specified object in any scanning image. According to the technical scheme, the accuracy of image segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular, to an image processing method, an image processing device, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] Image segmentation is a classic problem in computer vision research and has become a hot topic in the field of image understanding. In related technologies, image segmentation can be performed based on two-dimensional convolutional neural networks (2D CNN) and three-dimensional convolutional neural networks (3D CNN). Among them, the image features extracted based on 2D CNN do not contain depth information, resulting in low accuracy of image segmentation; and in the process of extracting image features, although 3D CNN can use the depth information of the image, it is difficult to accurately capture the corresponding regional edge details for images with blurred regional edges such as medical images, resulting in insufficient accuracy of image segmentation. Therefore, how to improve the accuracy of image segmentation is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, an image processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of image segmentation.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Acquire a scanning image sequence obtained by performing layer-by-layer scanning of a spatial cross section of a specified object;

[0007] Performing initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of the any scanned image;

[0008] Predicting a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship characterizing a change in a cross-sectional edge from an adjacent image to a corresponding scanned image;

[0009] Based on the any scanned image and the inverse change relationship representing the change of the cross-sectional edge from the scanned image to the corresponding adjacent image, predicting an adjacent edge feature map of an adjacent image of the any scanned image;

[0010] A target image region presenting a cross section of the designated object in any one of the scanned images is determined according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising an acquisition unit, a processing unit and a determination unit, wherein:

[0012] The acquisition unit is used to acquire a scanning image sequence obtained by performing spatial cross-section layer-by-layer scanning on a designated object;

[0013] The processing unit is used to perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of any scanned image;

[0014] The processing unit is further used to predict a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship representing a change in a cross-sectional edge from an adjacent image to a corresponding scanned image;

[0015] The processing unit is further used to predict an adjacent edge feature map of an adjacent image of the any scanned image based on the any scanned image and an inverse change relationship representing a cross-sectional edge change from the scanned image to a corresponding adjacent image;

[0016] The determination unit is used to determine a target image region presenting a cross section of the designated object in any one of the scanned images according to the initial edge feature map, the reference edge feature map and the adjacent edge feature map.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a storage device for storing one or more computer programs, so that when the one or more computer programs are executed by the one or more processors, the electronic device implements the image processing method as described above.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the image processing method as described above.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the image processing method as described above.

[0020] In the technical solution provided in the embodiment of the present application, on the basis of initially obtaining the initial edge feature map of the scanned image through feature extraction processing, the reference edge feature map of the scanned image and the adjacent edge feature map of the adjacent image of the scanned image are further predicted through the positive and reverse change relationship of the cross-sectional edges between the scanned image and the adjacent image of the scanned image. Since the cross-sectional edges between the scanned image and its adjacent images have spatial correlation, the reference edge feature map and the adjacent edge feature map can provide richer edge feature information for the subsequent segmentation of the target image area, which is equivalent to capturing more cross-sectional edge details to assist in identifying the cross-sectional edges, so that more edge information can be used for accurate image segmentation in the subsequent process, which is conducive to improving the accuracy of image segmentation.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural schematic diagram of an image processing system provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 3 is a flowchart of another image processing method provided in an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the architecture of an image segmentation model provided in an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the architecture of another image segmentation model provided in an embodiment of the present application;

[0027] Figure 6 It is a flowchart of another image processing method provided in an embodiment of the present application;

[0028] Figure 7a is a schematic diagram of a training process provided in an embodiment of the present application;

[0029] Figure 7b It is a processing flow chart of an image segmentation model provided in an embodiment of the present application;

[0030] Figure 7c It is a general strategy diagram of an image segmentation model provided in an embodiment of the present application;

[0031] Figure 8 It is a flowchart of another image processing method provided in an embodiment of the present application;

[0032] Fig. 9 is a structural block diagram of an image processing device provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0038] Artificial Intelligence (AI) technology refers to the theories, methods, techniques and application systems that use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science; it mainly produces a new intelligent machine that can respond in a similar way to human intelligence by understanding the essence of intelligence, so that the intelligent machine has multiple functions such as perception, reasoning and decision-making. Accordingly, AI technology is a comprehensive discipline, which mainly includes several major directions such as computer vision technology (Computer Vision, CV), speech technology, natural language processing technology and machine learning / deep learning.

[0039] The so-called computer vision technology is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, locate and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0040] The research of various technologies in computer vision technology needs to be realized through machine learning. The so-called machine learning is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to make computer devices intelligent. The so-called machine learning is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computer devices simulate or realize human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Deep learning is a technology that uses deep neural network systems to perform machine learning. Machine learning / deep learning usually includes artificial neural networks, reinforcement learning (RL), supervised learning, unsupervised learning and other technologies. The so-called supervised learning refers to the processing method of using training samples with known categories (with labeled categories) to train models, and unsupervised learning refers to the processing method of using training samples with unknown categories (without being labeled) to train models.

[0041] Image segmentation is a classic problem in the research process of computer vision technology and a hot topic in the field of image understanding. In related technologies, image segmentation methods mainly include two categories, namely, segmentation methods based on two-dimensional convolutional neural networks (i.e., 2D models, using 2D CNN architecture) and segmentation methods based on three-dimensional convolutional neural networks (i.e., 3D models, using 3D CNN architecture).

[0042] Specifically, the segmentation method based on the 2D model has relatively small parameters and computational complexity, which can reduce the burden on hardware while retaining more complete spatial information in the two-dimensional image. But at the same time, the image features extracted by the segmentation method based on the 2D model also lack depth information, resulting in different image segmentation accuracies. The segmentation method based on the 3D model can simultaneously extract spatial correlation information between multiple frames of adjacent images, so that it can use depth information to provide richer visual features for the segmentation task. However, the segmentation method based on the 3D model has a large amount of computation, so it has high requirements for the running memory of the device; at the same time, although the segmentation method based on the 3D model can use depth information to improve the accuracy of image segmentation to a certain extent, for images with blurred regional edges such as medical images, it is still difficult to accurately capture the corresponding regional edge details, resulting in the image segmentation accuracy is still not high enough.

[0043] Based on this, the embodiment of the present application provides an image processing scheme, which, after acquiring a scanning image sequence obtained by scanning a specified object layer by layer in a spatial section, can obtain an initial edge feature map of any scanning image, a reference edge feature map, and an adjacent edge feature map of an adjacent image of any scanning image in the scanning image sequence, and an adjacent image of any scanning image, by means of initial edge feature extraction processing, characterizing the positive change relationship of the cross-sectional edge change from the adjacent image to the corresponding scanning image, and characterizing the reverse change relationship of the cross-sectional edge change from the scanning image to the corresponding adjacent image. Finally, the target image area presenting the cross section of the specified object in any scanning image can be determined based on the initial edge feature map, the reference edge feature map, and the adjacent edge feature map, thereby completing the segmentation of the cross section of the specified object from any scanning image.

[0044] Among them, since the scanning image sequence is obtained by scanning the spatial cross-sections of the specified object layer by layer, the cross-section edges of adjacent images in the scanning image sequence are spatially correlated, that is, the main difference between the cross-sections of the specified object contained in adjacent images in the scanning image sequence is at the cross-section edges.

[0045] In addition, the initial edge feature map of any scanned image can be obtained by performing initial edge feature extraction processing on any scanned image; at the same time, the reference edge feature map of any scanned image can be predicted based on the adjacent images of any scanned image and the above-mentioned positive change relationship; and the adjacent edge feature map of the adjacent images of any scanned image can be predicted based on any scanned image and the above-mentioned reverse change relationship.

[0046] It is not difficult to see that in addition to preliminarily obtaining the initial edge feature map of the scanned image through feature extraction processing, this solution will further predict the reference edge feature map of the scanned image and the adjacent edge feature map of the adjacent image of the scanned image through the positive and reverse change relationship of the cross-section edge between the scanned image and the adjacent image of the scanned image. Since the cross-section edges between the scanned image and its adjacent images have spatial correlation, the reference edge feature map and the adjacent edge feature map can provide richer edge feature information for the subsequent segmentation of the cross section.

[0047] It can be seen that, based on the scanned image and its adjacent images, this scheme obtains richer edge feature information that is spatially correlated with the cross-sectional edges in the scanned image through the reference edge feature map and the adjacent edge feature map predicted by the forward change relationship and the reverse change relationship, which is equivalent to capturing more cross-sectional edge details, thereby helping to improve the accuracy of image segmentation.

[0048] Based on the above image processing solution, the present application embodiment provides an image processing system, which can be seen in Figure 1 , Figure 1 The image processing system shown may include multiple terminal devices 101 and multiple servers 102. Among them, a communication connection is established between any terminal device and any server. For example, the terminal device 101 may include any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car, a medical imaging device, and a smart wearable device. A multimedia playback client, an image display client, a browser client, an information stream client, etc. can be run in the terminal device 101. The server 102 can be a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Any terminal device 101 and any server 102 can be directly or indirectly connected in communication via wired or wireless communication, and this application is not limited here.

[0049] In some embodiments, the above image processing method can be performed only by Figure 1 The terminal device 101 in the image processing system shown in the figure is executed, and the specific execution process is as follows: the terminal device 101 can first obtain a scanning image sequence obtained by scanning the spatial section layer by layer of the specified object. Then, the terminal device 101 can perform initial edge feature extraction processing on any scanning image in the scanning image sequence to obtain an initial edge feature map of any scanning image. At the same time, the terminal device 101 can predict the reference edge feature map of any scanning image based on the adjacent images of any scanning image and the positive change relationship that characterizes the change of the section edge from the adjacent image to the corresponding scanning image. And, the terminal device 101 can predict the adjacent edge feature map of the adjacent image of any scanning image based on any scanning image and the reverse change relationship that characterizes the change of the section edge from the scanning image to the corresponding adjacent image. Finally, the terminal device 101 can determine the target image area in any scanning image that presents the section of the specified object based on the initial edge feature map, the reference edge feature map, and the adjacent edge feature map.

[0050] Optionally, the terminal device 101 may also output the target image area. Furthermore, the method of outputting the target image area may be: the terminal device 101 performs color change processing on the target image area in any scanned image to obtain a processed scanned image; and finally outputs the processed scanned image. The method of outputting the target image area may also be: the terminal device 101 marks the area edge of the target image area in any scanned image, and outputs the marked scanned image. Optionally, the terminal device 101 may also output the target image area in other ways, which are not limited to this.

[0051] Optionally, the above image processing method may also be performed only by Figure 1 The server 102 in the image processing system shown in the figure executes, and the specific execution process thereof can refer to the specific execution process of the terminal device 101, which will not be repeated here.

[0052] In some other embodiments, the above image processing method may be run in an image processing system, which may include a terminal device and a server. Figure 1 The terminal device 101 and the server 102 included in the image processing system shown in the figure are jointly completed, and the specific execution process is as follows: the terminal device 101 can perform spatial cross-section layer-by-layer scanning on the specified object to obtain a scanned image sequence; then, the terminal device 101 can send the scanned image sequence to the server 102. After receiving the scanned image sequence, the server 102 can first perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of any scanned image. At the same time, the server 102 can predict the reference edge feature map of any scanned image based on the adjacent images of any scanned image and the positive change relationship representing the cross-section edge change from the adjacent image to the corresponding scanned image. And, the server 102 can predict the adjacent edge feature map of the adjacent image of any scanned image based on any scanned image and the reverse change relationship representing the cross-section edge change from the scanned image to the corresponding adjacent image. Then, the server 102 can determine the target image area of ​​the cross section of the specified object in any scanned image based on the initial edge feature map, the reference edge feature map and the adjacent edge feature map.

[0053] Optionally, the server 102 may send the target image area to the terminal device 101. Optionally, the server 102 may also perform color change processing on the target image area in any scanned image to obtain a processed scanned image, and finally the server 102 may send the processed scanned image to the terminal device 101. Optionally, the server 102 may also annotate the region edge of the target image area in any scanned image, and send the annotated scanned image to the terminal device 101. Optionally, after receiving the target image area or scanned image sent by the server 102, the terminal device 101 may also output the corresponding target image area or scanned image.

[0054] It should be noted that the embodiments of the present application can process images that require image segmentation and can be applied to various scenarios, including but not limited to cloud technology, AI (Artificial Intelligence), smart medical care, smart home, assisted driving and other scenarios. It can also be used in medical applications, conference applications, social applications, shopping applications and other applications that can transmit or view images, without limitation.

[0055] Specifically, the technical solution of the embodiment of the present application can be applied to the smart medical scene, such as setting the terminal device as a nuclear magnetic resonance instrument, the designated object can be a human brain, the cross section of the designated object can be a glioma in the brain, and the server can be a server that provides image processing services for the terminal device in the smart medical scene. Specifically, the nuclear magnetic resonance instrument can scan the brain layer by layer in spatial cross section to obtain a brain image sequence; then, the nuclear magnetic resonance instrument can send the brain image sequence to the server.

[0056] After receiving the brain image sequence, the server can perform initial edge feature extraction processing on any brain image in the brain image sequence to obtain an initial edge feature map of any brain image; at the same time, the server can predict a reference edge feature map of any brain image based on the adjacent images of any brain image and the positive change relationship representing the glioma edge change from the adjacent image to the corresponding brain image; and the server can predict an adjacent edge feature map of an adjacent image of any brain image based on any brain image and the reverse change relationship representing the glioma edge change from the brain image to the corresponding adjacent image. Then, the server can determine the target image area showing brain glioma in any brain image based on the initial edge feature map of any brain image, the reference edge feature map and the adjacent edge feature map of the adjacent image of any brain image.

[0057] Optionally, the server may also send the target image region showing the brain glioma in any brain image to the terminal device used by the doctor, and the terminal device used by the doctor may highlight the target image region in any brain image.

[0058] For another example, the technical solution of the embodiment of the present application is applied to a conference application, the terminal device may be a terminal device running a social application, the designated object may be the patient's lungs, the cross section of the designated object may be a lung lesion area, and the server may be a server that provides image processing services for the conference application. The object using the conference application may be a doctor, who may conduct remote consultations, remote teaching, etc. through the conference application.

[0059] Specifically, the doctor can input the spatial cross-sectional scan of the patient's lungs layer by layer in the conference interface of the conference application to obtain a lung image sequence. Then, the terminal device can send the lung image sequence to the server.

[0060] After receiving the lung image sequence, the server can perform initial edge feature extraction processing on any lung image in the lung image sequence to obtain the initial edge feature map of any lung image; at the same time, the server can predict the reference edge feature map of any lung image based on the adjacent images of any lung image and the positive change relationship representing the edge change of the lung lesion area from the adjacent image to the corresponding brain image; and the server can predict the adjacent edge feature map of any lung image's adjacent image based on any lung image and the reverse change relationship representing the edge change of the lung lesion area from the lung image to the corresponding adjacent image. Then, the server can determine the target image area presenting the lung lesion area in any lung image based on the initial edge feature map of any lung image, the reference edge feature map and the adjacent edge feature map of any lung image's adjacent image.

[0061] Optionally, the server may also send a target image area presenting a lung lesion area in any lung image to terminal devices used by participants in the above conference interface, and the corresponding terminal device may highlight the target image area in any lung image.

[0062] It should be noted that in the specific implementation of the present application, if the scanned images, scanned image sequences and other related image data are related to the object, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of the relevant image data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0063] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0064] like Figure 2 As shown, Figure 2 is a flowchart of an image processing method according to an embodiment of the present application, which can be applied to Figure 1The image processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the terminal device as an example for explanation, wherein the image processing method may include S201 to S204, which are described in detail as follows:

[0065] S201, obtaining a scanning image sequence obtained by performing spatial cross-sectional layer-by-layer scanning on a designated object.

[0066] In the embodiment of the present application, the designated object may be a three-dimensional object. The scanning image sequence may include multiple scanning images, each of which includes a tomographic section of the designated object, and the image region representing the target object in the tomographic section is the section of the designated object; adjacent scanning images include adjacent sections and have spatial correlation.

[0067] For example, if the designated object is a brain and the target object is a glioma in the brain, the scanned image includes a cross-sectional view of the brain, and the cross-sectional view of the designated object refers to an image region representing the glioma in the cross-sectional view of the brain.

[0068] Optionally, since a scanned image often has a background and a foreground, the foreground in the scanned image refers to the area of ​​the tomographic section representing the specified object, and therefore the tomographic section of the specified object can be the same as the image area representing the target object (i.e., the section of the specified object); for example, the tomographic section of the specified object is the image area representing the brain, since the tomographic section of the specified object is the same as the section of the specified object, the section of the specified object is also the tomographic section of the brain.

[0069] In one embodiment, the spatial cross-section of the specified object can be scanned layer by layer by using instruments such as a nuclear magnetic resonance apparatus and a computer tomography scanner, or by using algorithms such as filtered back projection and iterative reconstruction, which are not limited here.

[0070] Optionally, the specific method of scanning the space section layer by layer may be: scanning the space section layer by layer of the designated object in a preset orientation sequence to obtain a scan image sequence, wherein the preset orientation sequence may be from top to bottom or from left to right, which is not limited here.

[0071] Optionally, the specific method of layer-by-layer scanning of the spatial section can also be: performing layer-by-layer scanning of the spatial section on the preset part of the specified object to obtain a scanned image sequence. The preset part of the specified object can be artificial, or a part of the position of the specified object set by the terminal device or server in the above-mentioned image processing system; if the specified object is a cube with a height of 10 cm, the preset part of the specified object can be a part of the cube with a height of 5 to 6 cm. Optionally, layer-by-layer scanning of the spatial section can also be performed by other methods, which are not limited here.

[0072] S202: Perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of any scanned image.

[0073] In an embodiment of the present application, the specific method of initial edge feature extraction processing can be: through the initial segmentation network in the image segmentation model, initial edge feature extraction processing is performed on any scanned image to obtain an initial edge feature map of any scanned image.

[0074] Optionally, the training process of the initial segmentation network in the image segmentation model can be: first, obtain a training image sequence obtained by scanning the spatial section of the training object layer by layer, and a training image area presenting the section of the training object in each training image included in the training image sequence; then, use the initial segmentation network in the image segmentation model to be trained to perform initial edge feature extraction processing on each training image in the training image sequence to obtain an initial edge feature map of each training image; thereafter, use the initial segmentation network in the image segmentation model to be trained to perform classification and prediction processing on the initial edge feature map of each training image to obtain a predicted image area presenting the section of the training object in the corresponding training image; finally, based on the training image area and the predicted image area corresponding to each training image, the initial segmentation network in the image segmentation model to be trained is trained.

[0075] The training image sequence includes a plurality of training images containing cross sections of the training object, each training image includes a cross section of the training object, and the image region representing the reference object in the cross section is the cross section of the specified object; the cross sections contained in adjacent training images are adjacent and have spatial correlation. For a specific description of the reference object, please refer to the specific description of the target object in step S201, which will not be repeated here.

[0076] In a specific implementation, the network architecture of the initial segmentation network can be a fully convolutional network (FCN) such as AlexNet (a convolutional neural network designed by the winner of the 2012 ImageNet competition and his students), Visual Geometry Group Networks (VGGNet), etc.; it can also be a neural network with an encoder-decoder architecture such as U-Net (a semantic segmentation network), or other network architectures that can be used for image segmentation, which are not limited here.

[0077] It should be noted that, when the network architecture of the initial segmentation network is an encoder-decoder architecture, the initial edge feature map of each training image used for classification prediction processing may include a feature map output by the encoder in the initial segmentation network.

[0078] S203, predicting a reference edge feature map of any scanned image based on adjacent images of any scanned image and a positive change relationship characterizing a cross-sectional edge change from an adjacent image to a corresponding scanned image.

[0079] In an embodiment of the present application, the positive change relationship can be summarized based on the law of cross-sectional edge change from adjacent images to each training image in multiple training images. For example, the positive change relationship can be expressed by a corresponding function; it can also be mapped to a network parameter in a corresponding neural network, which is not limited.

[0080] In one embodiment, the forward change relationship can be mapped to a network parameter in a forward reconstruction network included in the image segmentation model. Then, based on the adjacent images of any scanned image and the forward change relationship characterizing the change of the cross-sectional edge from the adjacent image to the corresponding scanned image, a specific method for predicting the reference edge feature map of any scanned image can be: the adjacent images of any scanned image are processed through the forward reconstruction network in the image segmentation model to predict the reference edge feature map of any scanned image.

[0081] Furthermore, the training process of the forward reconstruction network can be: first, obtain a training image sequence; then, through the forward reconstruction network in the image segmentation model to be trained, process the adjacent images of each training image in the training image sequence, and predict and generate the reconstructed image of each training image; thereafter, the forward reconstruction network in the image segmentation model to be trained can be trained based on each training image and the reconstructed image of each training image.

[0082] Among them, based on each training image and the reconstructed image of each training image, the specific method of training the forward reconstruction network in the image segmentation model to be trained can be: training the forward reconstruction network in the image segmentation model to be trained in the direction of reducing the error between the reconstructed image of each training image and each training image.

[0083] In a specific implementation, the network architecture of the forward reconstruction network can be a deep convolutional generative adversarial network, an encoder-decoder structure, etc., without limitation.

[0084] S204: predicting an adjacent edge feature map of an adjacent image of any scanned image based on any scanned image and an inverse change relationship characterizing a cross-sectional edge change from the scanned image to a corresponding adjacent image.

[0085] In an embodiment of the present application, the inverse change relationship can be summarized based on the law of cross-sectional edge change between each training image and adjacent images in multiple training images. For example, the inverse change relationship can be expressed by a corresponding function; it can also be mapped to a network parameter in a corresponding neural network, which is not limited.

[0086] In one embodiment, the inverse change relationship can be mapped to a network parameter in an inverse reconstruction network included in the image segmentation model. Then, based on any scanned image and the inverse change relationship that characterizes the change of the cross-sectional edge from the scanned image to the corresponding adjacent image, a specific method for predicting an adjacent edge feature map of an adjacent image of any scanned image can be: by processing the adjacent image of any scanned image through the inverse reconstruction network in the image segmentation model, the adjacent image of any scanned image is predicted to obtain an adjacent edge feature map of an adjacent image of any scanned image.

[0087] Furthermore, the training process of the inverse reconstruction network can be: first, obtain a training image sequence; then, process each training image in the training image sequence through the inverse reconstruction network in the image segmentation model to be trained, and predict and generate reconstructed images of adjacent images of each training image; thereafter, the inverse reconstruction network in the image segmentation model to be trained can be trained based on the adjacent images of each training image and the reconstructed images of the adjacent images of each training image.

[0088] Among them, based on the adjacent images of each training image and the reconstructed images of the adjacent images of each training image, the specific method of training the inverse reconstruction network in the image segmentation model to be trained can be: training the inverse reconstruction network in the image segmentation model to be trained in the direction of reducing the error between the reconstructed images of the adjacent images of each training image and the adjacent images of each training image.

[0089] In a specific implementation, the network architecture of the reverse reconstruction network can be a deep convolutional generative adversarial network, an encoder-decoder, etc., without limitation.

[0090] S205: Determine a target image region presenting a cross section of a designated object in any scanned image according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map.

[0091] In an embodiment of the present application, a specific method for determining the target image area based on the initial edge feature map, the reference edge feature map, and the adjacent edge feature map can be: performing feature fusion on the initial edge feature map, the reference edge feature map, and the adjacent edge feature map to obtain a fused feature map; and then performing classification and prediction processing on the fused feature map to obtain a target image area presenting a cross-section of a specified object in any scanned image.

[0092] The specific method of feature fusion may be: adding the initial edge feature map, the reference edge feature map and the adjacent edge feature map at the feature channel level to obtain a fused feature map. Optionally, the initial edge feature map, the reference edge feature map and the adjacent edge feature map may be fused at multiple levels to determine the target image area showing the cross section of the specified object in any scanned image. Optionally, feature fusion may also be performed in other ways, which are not limited here.

[0093] In addition, the specific method of classification prediction processing can be: calling the pre-selected activation function, classifying the fused feature map, predicting the probability that the object presented by the corresponding pixel in the fused feature map is the cross section of the specified object, thereby obtaining the target image area in any scanned image. Among them, the pre-selected activation function can be any function among multiple functions such as hyperbolic tangent function (i.e. Tanh function), S-shaped growth curve function (i.e. sigmoid function), linear rectification function (i.e. Relu function), parameter corrected linear function (i.e. PReLU function), etc., and there is no limitation on this.

[0094] In the embodiment of the present application, on the basis of initially obtaining the initial edge feature map of the scanned image through feature extraction processing, the reference edge feature map of the scanned image and the adjacent edge feature map of the adjacent image of the scanned image are further predicted through the change relationship of the cross-section edge between the scanned image and the adjacent image of the scanned image in the positive and reverse directions. Since the cross-section edges between the scanned image and its adjacent images have spatial correlation, the reference edge feature map and the adjacent edge feature map can provide richer edge feature information for the subsequent segmentation of the cross section.

[0095] It can be seen that, based on the initial edge feature map, the embodiment of the present application will also predict the reference edge feature map and the adjacent edge feature map through the forward change relationship and the reverse change relationship, so as to obtain richer edge feature information that has spatial correlation with the cross-sectional edges in the scanned image, which is equivalent to capturing more cross-sectional edge details, and is facilitating the subsequent use of more edge information for accurate image segmentation, thereby achieving the purpose of improving the accuracy of image segmentation.

[0096] In one embodiment of the present application, another image processing method is provided, which can be applied to Figure 1 The image processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 3 FIG. 4 is a flow chart showing another image processing method. Figure 2 This paper expands on the method shown in .

[0097] Among them, S301 to S307 are described in detail as follows:

[0098] S301, obtaining a scanning image sequence obtained by performing spatial cross-sectional layer-by-layer scanning on a designated object.

[0099] In the embodiment of the present application, the specific implementation of step S301 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0100] S302: Perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of any scanned image.

[0101] In the embodiment of the present application, the specific implementation of step S302 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0102] S303: predict a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship characterizing a cross-sectional edge change from an adjacent image to a corresponding scanned image.

[0103] In the embodiment of the present application, the adjacent images of any scanned image may include the previous frame image and the next frame image of any scanned image in the scanned image sequence. Specifically, the previous frame image may include n frames of images before any scanned image, and the next frame image may include m frames of images after any scanned image; wherein n and m are both positive integers. The adjacent images of the training image may refer to the specific explanation of the adjacent images of any scanned image, which will not be repeated later.

[0104] Optionally, if any scanned image is the first frame image in a scanned image sequence or the last frame image in a scanned image sequence, then a pure background image is added before the first frame image as the previous frame image of the first frame image, or a pure background image is added after the last frame image as the next frame image of the last frame image.

[0105] In one embodiment, based on the adjacent images of any scanned image and the positive change relationship that characterizes the cross-sectional edge change from the adjacent images to the corresponding scanned image, a specific method for predicting the reference edge feature map of any scanned image can be: splicing the previous frame image and the subsequent frame image of any scanned image to obtain a target spliced ​​image; based on the target spliced ​​image and the positive change relationship, obtaining the reference edge feature map of any scanned image.

[0106] Optionally, since step S203 mentions that the positive change relationship can be mapped to the network parameters in the forward reconstruction network included in the image segmentation model, and the network architecture of the forward reconstruction network can be an encoder-decoder structure; and after the encoder outputs the corresponding encoding feature map, the corresponding decoder will decode the encoding feature map to obtain the corresponding decoding feature map. Therefore, the reference edge feature map can include a reference encoding feature map and a reference decoding feature map.

[0107] Then, based on the target spliced ​​image and the forward change relationship, the specific method of obtaining the reference edge feature map of any scanned image can be: calling the forward reconstruction network in the image segmentation model, downsampling the target spliced ​​image to obtain a reference low-dimensional feature map; performing high-dimensional convolution processing on the reference low-dimensional feature map to obtain a reference encoding feature map; decoding the reference encoding feature map to obtain a reference decoding feature map.

[0108] S304: Based on any scanned image and the inverse change relationship that characterizes the change of the cross-sectional edge from the scanned image to the corresponding adjacent image, predict an adjacent edge feature map of an adjacent image of any scanned image.

[0109] In the embodiment of the present application, the specific implementation of step S304 can refer to the specific implementation of step S204 in the above embodiment regarding reverse reconstruction of the network, which will not be repeated here.

[0110] S305: Fusing the reference edge feature map and the adjacent edge feature map to obtain a fused edge feature map.

[0111] In an embodiment of the present application, a specific method of feature fusion may be: adding a reference edge feature map and an adjacent edge feature map at a feature channel level to obtain a fused edge feature map. Optionally, a method of feature fusion may also be: mapping a reference edge feature map and an adjacent edge feature map into feature maps of the same size, and cross-multiplying the two mapped feature maps to obtain a fused edge feature map. Optionally, feature fusion may also be performed in other ways, which are not limited here.

[0112] In one embodiment, since the network architectures of the forward reconstruction network and the reverse reconstruction network mentioned in step S203 and step S204 can both be encoder-decoder structures, after the encoder outputs the corresponding encoding feature map, the corresponding decoder will decode the encoding feature map to obtain the corresponding decoding feature map.

[0113] Therefore, the reference edge feature map may include a reference encoding feature map and a reference decoding feature map, and the adjacent edge feature map may include an adjacent encoding feature map and an adjacent decoding feature map, wherein any decoding feature map is obtained by decoding the corresponding encoding feature map; at the same time, the fused edge feature map may include a fused encoding feature map and a fused decoding feature map.

[0114] Then, the specific process of performing feature fusion on the reference edge feature map and the adjacent edge feature map to obtain the fused edge feature map can be: performing feature fusion on the reference encoding feature map and the adjacent encoding feature map to obtain the fused encoding feature map; performing feature fusion on the reference decoding feature map and the adjacent decoding feature map to obtain the fused decoding feature map.

[0115] S306, performing feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain a target feature map of any scanned image.

[0116] In an embodiment of the present application, the size of the target feature map is the same as the size of any scanned image; the specific method of feature residual fusion processing can be: through the multi-feature fusion network in the image segmentation model, feature residual fusion processing is performed on the fused edge feature map and the initial edge feature map to obtain the target feature map of any scanned image.

[0117] Optionally, the multi-feature fusion network may include a feature merging layer, a residual convolution layer, a residual pooling layer and a residual addition layer connected in sequence; then, the specific method of performing feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain the target feature map may be: inputting the fused edge feature map and the initial edge feature map into the multi-feature fusion network, so as to merge the fused edge feature map and the initial edge feature map through the feature merging layer in the multi-feature fusion network to obtain the added feature map; then, performing residual convolution processing on the added feature map through the residual convolution layer, and performing residual pooling processing on the feature map output by the residual convolution layer through the residual pooling layer to obtain the pooled feature map; finally, performing feature addition of the pooled feature map and the initial edge feature map through the residual addition layer to obtain the target feature map.

[0118] Optionally, the network structure of the multi-feature fusion network may also be other network structures that can utilize the feature map output by the previous network layer for residual connection, and there is no limitation on this.

[0119] In one embodiment, since the network architecture of the initial segmentation network mentioned in step S202 may also be an encoder-decoder architecture; therefore, the initial edge feature map may include an initial encoding feature map and an initial decoding feature map.

[0120] Then, the specific method of feature residual fusion processing can also be: perform residual merging processing on the fused coding feature map and the initial coding feature map to obtain a residual coding feature map; perform decoding processing on the residual coding feature map to obtain a residual decoding feature map; perform residual merging processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain a target feature map.

[0121] Among them, the specific method of performing residual merging processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain the target feature map can be: performing cross multiplication processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain the processed feature map; performing feature convolution processing on the processed feature map to obtain the convolved feature map; performing feature fusion on the convolved feature map and the initial encoding feature map to obtain the target feature map.

[0122] Similarly, the specific method of performing residual merging processing on the fused coding feature map and the initial coding feature map to obtain the residual coding feature map can be: cross-multiplying the fused coding feature map and the initial coding feature map to obtain the corresponding feature map; performing feature convolution processing on the corresponding feature map to obtain the feature map after feature convolution processing; and performing feature fusion on the feature map after feature convolution processing and the initial coding feature map to obtain the residual coding feature map.

[0123] S307: Perform pixel classification processing according to the target feature map to obtain a prediction probability matrix of any scanned image.

[0124] In the embodiment of the present application, each element in the prediction probability matrix is ​​used to characterize the probability that the object presented by the corresponding pixel in the target feature map is a cross-section of a specified object.

[0125] Since the size of the target feature map is the same as that of any scanned image, each element in the target feature map has a one-to-one correspondence with each pixel in any scanned image. Then, the specific method of pixel classification processing can be: through a preset activation function, each element in the target feature map is activated and mapped to obtain the probability that the object presented by the corresponding pixel in the target feature map is a cross-section of a specified object; finally, according to each probability, a prediction probability matrix of any scanned image is obtained.

[0126] In a specific implementation, the preset activation function can be any one of a variety of functions, such as a hyperbolic tangent function (ie, Tanh function), an S-shaped growth curve function (ie, sigmoid function), a linear rectification function (ie, Relu function), a parameter corrected linear function (ie, PReLU function), etc., without limitation.

[0127] S308 . Based on the prediction probability matrix of any scanned image, perform cross-sectional segmentation on any scanned image to obtain a target image region.

[0128] In an embodiment of the present application, based on the prediction probability matrix of any scanned image, any scanned image is segmented into sections, and the specific method for obtaining the target image area can be: if the probability represented by any element in the prediction probability matrix of any scanned image is greater than or equal to a preset probability threshold, then the pixel corresponding to the any element in any scanned image is determined to be the target pixel; finally, based on each target pixel, the target image area in any scanned image is determined.

[0129] For specific implementation, please refer to the attached Figure 4 , shows a schematic diagram of the architecture of an image segmentation model. Figure 4 As shown, the image segmentation model 401 includes an initial segmentation network 402, a multi-feature fusion network 403, and a bidirectional reconstruction network including a forward reconstruction network 404 and a reverse reconstruction network 405.

[0130] Among them, the initial segmentation network 402 includes an initial encoder and an initial decoder connected in sequence; the multi-feature fusion network 403 includes a first residual merging module, a decoding module, a second residual merging module and an activation function layer 406; the forward reconstruction network 404 includes a splicing layer, a forward encoder and a forward decoder connected in sequence; and the reverse reconstruction network 405 includes a reverse encoder and a reverse decoder connected in sequence.

[0131] like Figure 4 As shown, the target frame scan image X in the scan image sequence N is scanned i , target frame scan image X i The previous frame image X i-1 And the target frame scan image X i The next frame image X i-1 Input image segmentation model 401; wherein N is a positive integer, i is a positive integer, and i is less than or equal to the total number of images N in the scanned image sequence.

[0132] Then, the target frame scan image X can be i Input the initial segmentation network, and the initial encoder scans the target frame image X i The initial encoding feature map is processed to obtain an initial encoding feature map; and the initial decoder processes the initial encoding feature map to obtain an initial decoding feature map.

[0133] In addition, the previous frame image X i-1 And the next frame image X i-1 The input is the forward reconstruction network 404, and the previous frame image X is transformed into i-1 And the next frame image X i-1 The spliced ​​images are spliced ​​and input into the forward encoder for processing to obtain a reference coding feature map; then, the reference coding feature map can be processed by the forward decoder to obtain a reference decoding feature map.

[0134] In addition, the target frame scan image X is also i Input the reverse reconstruction network 405, and the reverse encoder scans the target frame image X i Processing is performed to obtain an adjacent coding feature map; and the adjacent coding feature map is processed by an inverse decoder to obtain an adjacent decoding feature map.

[0135] like Figure 4 As shown, after the reverse encoder outputs the adjacent coding feature map, the adjacent coding feature map can be fused with the reference coding feature map output by the forward encoder to obtain a fused coding feature map.

[0136] After obtaining the fused coding feature map, the fused coding feature map and the initial coding feature map output by the initial encoder can be subjected to residual merging processing by the first residual merging module in the multi-feature fusion network 403 to obtain a residual coding feature map; the residual coding feature map is then processed by the decoding module to obtain a residual decoding feature map.

[0137] Afterwards, if Figure 4As shown in FIG. 1 , after the reverse decoder outputs the adjacent decoding feature map, the adjacent decoding feature map can be feature fused with the reference decoding feature map output by the forward decoder to obtain a fused decoding feature map. Then, the second residual merging module performs residual merging processing on the fused decoding feature map, the residual coding feature map, and the initial decoding feature map output by the initial decoder to obtain the target frame scan image X i Target feature map.

[0138] Finally, if Figure 4 As shown, the activation function layer 406 scans the target frame image X i The target feature map is activated and mapped to obtain the target frame scanned image X with the target image region 407 marked, which is finally output by the image segmentation model 401. i .

[0139] In a possible implementation manner, since the encoder may include a plurality of encoding layers connected in sequence, the decoder may include a plurality of decoding layers connected in sequence.

[0140] Then, the forward encoder may include a plurality of forward encoding layers connected in sequence, and the forward decoder may include a plurality of forward decoding layers connected in sequence. Therefore, the number of reference encoding feature maps may include a plurality, and the number of reference decoding feature maps may also include a plurality; any reference encoding feature map is output after encoding the reference encoding feature map output by the previous forward encoding layer through the corresponding forward encoding layer; any reference decoding feature map is output after decoding the reference decoding feature map output by the previous forward decoding layer through the corresponding forward decoding layer.

[0141] Similarly, the inverse encoder may include multiple inverse encoding layers connected in sequence, and the inverse decoder may include multiple inverse decoding layers connected in sequence. Therefore, the number of adjacent encoding feature maps may include multiple, and the number of adjacent decoding feature maps may also include multiple.

[0142] Furthermore, the initial encoder may include multiple initial encoding layers connected in sequence, and the inverse decoder may include multiple initial decoding layers connected in sequence. Therefore, the number of initial encoding feature maps may include multiple, and the number of initial decoding feature maps may also include multiple.

[0143] Further, each forward decoding layer in the multiple forward decoding layers, each reverse decoding layer in the multiple reverse decoding layers, and each initial decoding layer in the multiple initial decoding layers may have a corresponding relationship; and each forward coding layer in the multiple forward coding layers, each reverse coding layer in the multiple reverse coding layers, and each initial coding layer in the multiple initial coding layers may have a corresponding relationship. The corresponding relationship may be one-to-one or one-to-many, which is not limited here and will not be described in detail later.

[0144] Then, the number of fused coding feature maps and fused decoding feature maps may also include multiple ones; the specific method of performing feature fusion on the reference coding feature map and the adjacent coding feature map to obtain the fused coding feature map may be: performing feature fusion on each reference coding feature map and the adjacent coding feature map output by the reverse coding layer corresponding to the forward coding layer that outputs each reference coding feature map to obtain each fused coding feature map.

[0145] Similarly, the specific method of performing feature fusion on the reference decoding feature map and the adjacent decoding feature map to obtain the fused decoding feature map can be: performing feature fusion on each reference decoding feature map and the adjacent decoding feature map output by the reverse decoding layer corresponding to the forward decoding layer that outputs each reference decoding feature map to obtain each fused decoding feature map.

[0146] Furthermore, the multi-feature fusion network may include multiple residual merging modules, and a decoding module is connected between any two residual merging modules; there is a corresponding relationship between the residual merging module and the encoding layer or decoding layer in the initial segmentation network, the encoding layer or decoding layer in the forward reconstruction network, and the encoding layer or decoding layer in the reverse reconstruction network.

[0147] Then, the specific process of feature residual fusion can be: through each residual merging module, the feature map output by the previous residual merging module of each residual merging module, the feature map output by the initial coding layer or the initial decoding layer corresponding to each residual merging module, and the corresponding fused coding feature map or fused decoding feature map are subjected to residual merging processing to obtain the feature map output by each residual merging module.

[0148] In practical applications, please refer to the attached Figure 5 , shows a schematic diagram of the architecture of another image segmentation model. Figure 5 As shown, the image segmentation model 501 includes an initial segmentation network 502, a multi-feature fusion network 503, and a bidirectional reconstruction network including a forward reconstruction network 504 and a reverse reconstruction network 505.

[0149] Among them, the initial segmentation network 502 includes three low-dimensional coding layers, two high-dimensional coding layers and three decoding layers connected in sequence. The multi-feature fusion network 503 includes four residual merging modules and three decoding layers, and a decoding layer is connected between every two residual merging modules. The forward reconstruction network 504 includes a splicing layer, three low-dimensional coding layers, two high-dimensional coding layers and three decoding layers. The reverse reconstruction network 505 includes a reverse reconstruction network 505.

[0150] Specifically, the low-dimensional coding layer may include a normalization layer, an activation function layer, a convolution layer, and a downsampling layer connected in sequence. Optionally, the low-dimensional coding layer may also include an attention layer, wherein the attention layer may weight the contribution of each element in the feature map to enhance effective information (such as information about the edge of the section) and weaken the interference of invalid information, thereby improving the efficiency and accuracy of the image segmentation model. The attention layer will not be described in detail later.

[0151] The high-dimensional coding layer may include a normalization layer, an activation function layer, and a high-dimensional convolution layer, through which the abstract visual feature information of the scanned image in the high-dimensional feature space may be extracted. Optionally, the high-dimensional coding layer may also include an attention layer.

[0152] The decoding layer may include a normalization layer, an activation function layer, and an upsampling layer; optionally, the decoding layer may also include an attention layer. The decoding layer is used to restore the size of the low-resolution feature map output by the encoding layer to the same size as the scanned image. The residual merging module may include a cross-product layer, a convolution layer, an activation function layer, and a feature merging layer.

[0153] Optionally, in order to compensate for the details lost by the encoder during downsampling, the initial segmentation network adds skip connections between the encoding layer and the decoding layer with the same resolution. The skip connection can pass the low-level edge details extracted by the encoding layer to the decoding layer, so that the decoding layer can grasp rich multi-scale information when reconstructing the low-resolution feature map.

[0154] It should be noted that the network structures of the encoding layers, decoding layers, and residual merging modules in the image segmentation model may be the same or different, and the network parameters may be the same or different, and there is no limitation on this.

[0155] like Figure 5 As shown, the feature map output by the last high-dimensional coding layer in the forward reconstruction network 504 can be fused with the feature map output by the last high-dimensional coding layer in the reverse reconstruction network 505 to obtain a fused coding feature map input to the first residual merging module in the multi-feature fusion network 503; then, the feature map output by the last high-dimensional coding layer in the initial segmentation network 502 is residually merged with the fused coding feature map through the first residual merging module to obtain a feature map input to the first decoding layer in the multi-feature fusion network 503.

[0156] At the same time, the feature map output by the first decoding layer in the forward reconstruction network 504 can be fused with the feature map output by the first decoding layer in the reverse reconstruction network 505 to obtain a fused decoding feature map input to the second residual merging module in the multi-feature fusion network 503.

[0157] Afterwards, the feature map output by the first decoding layer in the multi-feature fusion network 503, the corresponding fused decoding feature map, and the decoding feature map output by the first decoding layer in the initial segmentation network 502 are subjected to residual merging processing by the second residual merging module to obtain the feature map input to the second decoding layer in the multi-feature fusion network 503. Similarly, the specific implementation of the subsequent third residual merging module and the fourth residual merging module can refer to the specific implementation of the first residual merging module and the second residual merging module, which will not be described in detail here. Finally, the feature map output by the last residual merging module is processed by the activation function layer 506 to obtain the target frame scan image X with the target image area 507 marked. i .

[0158] In the embodiment of the present application, on the basis of initially obtaining the initial edge feature map of the scanned image through the initial edge feature extraction process, the reference edge feature map of the scanned image and the adjacent edge feature map of the adjacent image of the scanned image are further predicted through the forward reconstruction network and the reverse reconstruction network. Since the cross-sectional edges between the scanned image and its adjacent images have spatial correlation, the reference edge feature map and the adjacent edge feature map can provide richer edge feature information for the subsequent segmentation of the cross section, which is equivalent to capturing more cross-sectional edge details, which is conducive to the subsequent use of more edge information for accurate image segmentation and is conducive to improving the accuracy of image segmentation.

[0159] In addition, the embodiment of the present application also uses feature residual fusion and multi-level fusion of the initial edge feature map, the reference edge feature map, and the adjacent edge feature map to fully extract and utilize the cross-section edge information, which is beneficial to further improve the accuracy of image segmentation. In addition, the embodiment of the present application can also add an attention layer in the relevant encoding layer and decoding layer to weaken the interference of invalid information and enhance the extraction of valid information through the attention layer, thereby further improving the efficiency and accuracy of the image segmentation model.

[0160] In one embodiment of the present application, another image processing method is provided, which can be applied to Figure 1 The image processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 6 FIG. 4 is a flow chart showing another image processing method. Figure 2 and Figure 3 This paper expands on the method shown in .

[0161] Among them, S601 to S608 are described in detail as follows:

[0162] S601 : Acquire a training image sequence obtained by performing spatial cross-section layer-by-layer scanning on a training object, and a training image region presenting a cross-section of the training object in each training image included in the training image sequence.

[0163] In the embodiment of the present application, the training object is a three-dimensional image, and the training object can be an object of the same type as the designated object; for example, if the designated object is a human brain, then the training object can also be a human brain, but the designated object and the training object can be brains of different people. Similarly, the cross section of the training object can be of the same type as the cross section of the designated object; for example, if the cross section of the training object can be a glioma in the human brain corresponding to the training object, then the cross section of the designated object can be a glioma in the human brain corresponding to the designated object.

[0164] The training image regions in each training image may be manually annotated; the specific implementation of the spatial tomography in step S601 may refer to the specific implementation of step S201 in the above embodiment, which will not be described in detail here.

[0165] S602: calling the image segmentation model to be trained, processing each training image, and obtaining a target prediction region of a cross section of the training object in each training image.

[0166] In an embodiment of the present application, the image segmentation model to be trained includes an initial segmentation network for initial feature extraction processing, a bidirectional reconstruction network including a forward reconstruction network and a reverse reconstruction network, and a fusion decoding network for determining a target prediction area. The forward reconstruction network is used to construct a forward change relationship by reconstructing each training image through adjacent images of each training image, and the reverse reconstruction network is used to construct a reverse change relationship by reconstructing adjacent images of each training image through each training image.

[0167] The network architectures of the initial segmentation network, the forward reconstruction network, and the reverse reconstruction network can refer to steps S203 to S204 in the above embodiment, and Figure 5 and Figure 6 The examples shown are not described in detail here.

[0168] Optionally, the initial segmentation network in the image segmentation model to be trained can be used not only to perform initial feature extraction processing on each training image to obtain an initial edge feature map of each training image, but also to perform classification prediction processing on the initial edge feature map of each training image to obtain a predicted image area presenting a cross-section of the training object in the corresponding training image.

[0169] Optionally, the adjacent images of each training image may include the previous frame image and the next frame image of each training image; then, the specific process of constructing the forward change relationship by reconstructing each training image through the adjacent images of each training image may be: calling the forward reconstruction network, splicing the previous frame image and the next frame image of each training image to obtain the training spliced ​​image of each training image; and predicting the reconstructed image of each training image based on the training spliced ​​image of each training image.

[0170] Optionally, the specific process of constructing the inverse change relationship by reconstructing adjacent images of each training image through each training image may be: calling the inverse reconstruction network to predict and obtain the previous frame reconstructed image and the next frame reconstructed image corresponding to each training image.

[0171] S603: Based on the target prediction area and the training image area corresponding to each training image, the image segmentation model to be trained is trained.

[0172] In an embodiment of the present application, based on the target prediction area and the training image area corresponding to each training image, the specific method of training the image segmentation model to be trained may be: based on the target prediction area and the training image area corresponding to each training image, determining the fourth loss value; training the image segmentation model to be trained in the direction of reducing the fourth loss value, thereby obtaining an image segmentation model.

[0173] Among them, the loss value can be calculated by one or more of a variety of loss functions such as cross entropy loss function, logarithmic loss function and square difference loss function, which will not be repeated later.

[0174] In order to further improve the accuracy of the trained image segmentation model, the network training of one or more networks among the initial segmentation network, the forward reconstruction network and the reverse reconstruction network can also be included in the training of the image segmentation model.

[0175] Optionally, referring to step S202 and step S602, the specific method of training the image segmentation model to be trained may also be: determining the fifth loss value based on the training image area and the predicted image area corresponding to each training image; and then, training the image segmentation model to be trained based on the fifth loss value and the fourth loss value.

[0176] The specific method of training the image segmentation model to be trained based on multiple loss values ​​may be: weighted addition of multiple loss values ​​to obtain a target loss value; and finally training the image segmentation model to be trained in a direction of reducing the target loss value; wherein the weight of each loss value is initially set manually, and can be changed in the process of model training to obtain the final weight of each loss value. The process of training based on multiple loss values ​​will not be described in detail later.

[0177] Optionally, referring to step S203 and step S602, the specific method of training the image segmentation model to be trained may also be: first determine a first loss value based on the previous frame reconstructed image and the previous frame image of each training image, and determine a second loss value based on the subsequent frame reconstructed image and the subsequent frame image of each training image; then, train the image segmentation model to be trained based on the first loss value, the second loss value, and the fourth loss value.

[0178] Optionally, referring to step S204 and step S602, the specific method of training the image segmentation model to be trained may also be: determining a third loss value based on the reconstructed image of each training image and each training image; and training the image segmentation model to be trained based on the third loss value and the fourth loss value.

[0179] Optionally, the specific method of training the image segmentation model to be trained may also be: training the image segmentation model to be trained based on the above-mentioned first loss value, second loss value, third loss value, fourth loss value and fifth loss value.

[0180] For specific implementation, please refer to the attached Figure 7a , which shows a schematic diagram of a training process. Figure 7a The training of the image segmentation model shown mainly includes two parts: data preprocessing and cross-section segmentation. The data preprocessing part includes data enhancement and data normalization; the cross-section segmentation part includes model construction and model verification.

[0181] In order to further expand the training data, data enhancement strategies such as left-right flipping, up-down flipping, and translation can be used to enhance the original training data. Compared with the data enhancement method based on generation, the enhanced data obtained by the above data enhancement strategy can retain a more realistic data distribution. Using the above enhanced data to train the model can make the model have the characteristics of flip invariance and translation invariance.

[0182] Due to the inconsistency of data collection agencies, there are differences in the training data collected by different agencies. This difference will affect the accuracy of image segmentation. Therefore, during the data preprocessing process, the collected training data can be normalized. Optionally, the maximum and minimum normalization method can be used for data normalization, that is, the pixel intensity of the original training image is converted to the range of [0,1] through a linear function. The maximum and minimum normalization method is suitable for cases where the values ​​are relatively concentrated. It has the characteristics of being lightweight, can significantly improve the data preprocessing speed, and reduce the loss of hardware caused by the normalization method.

[0183] Model construction refers to building an image segmentation model to be trained, which is composed of an initial segmentation network, a bidirectional reconstruction network including a forward reconstruction network and a reverse reconstruction network, and a multi-feature fusion network. Model validation refers to dividing the training image dataset into a training set, a validation set, and a test set. Optionally, since more data in the training set is required to train a more accurate model, the proportion of the training set in the training image dataset can be set to be greater than the proportion of the validation set and the proportion of the test set, or the proportion of the training set can be set to be greater than the sum of the proportions of the validation set and the test set.

[0184] In actual application, please refer to the attached Figure 7b , showing a processing flow chart of an image segmentation model. As shown in step 701, multiple brain magnetic resonance images (i.e., brain MRI data) can be obtained from BraTS 2018 (2018 brain tumor segmentation competition dataset), BraTS 2019 (2019 brain tumor segmentation competition dataset), and BraTS 2020 (2020 brain tumor segmentation competition dataset) as raw data.

[0185] The above three datasets all include training set, validation set and test set. The training set has the segmentation labels of brain glioma, which can be used as the segmentation standard of glioma to guide the training image segmentation model. The public brain MRI data information is shown in Table 1:

[0186] Table 1

[0187] Dataset Training set Validation set Test Set BraTS 2018 285 66 191 BraTS 2019 335 125 166 BraTS 2020 369 125 166

[0188] Then, as shown in step 702, data preprocessing, namely data normalization and data enhancement, may be performed on the original data.

[0189] The data after data preprocessing can be input into the image segmentation model to be trained. As shown in step 703 and step 704, the initial segmentation network in the image segmentation model to be trained can extract the glioma features in the brain magnetic resonance image (corresponding to the initial edge feature map); and the forward reconstruction network and the reverse reconstruction network in the image segmentation model to be trained can extract the edge features of the glioma in the brain magnetic resonance image (corresponding to the reference edge feature map and the adjacent edge feature map).

[0190] As shown in step 705, after the glioma features and edge features are obtained, the glioma features and edge features can be fused through the multi-feature fusion network in the image segmentation model to be trained; then, as shown in step 706, the fused features can be reconstructed through the decoding layer in the multi-feature fusion network, that is, the size of the features is restored to the size of the brain magnetic resonance image.

[0191] Finally, as shown in step 707 and step 708, the reconstructed features are activated by the activation function in the multi-feature fusion network, so that the glioma in the brain magnetic resonance image can be segmented.

[0192] For details, please see the attached Figure 7c , shows an overall strategy diagram of an image segmentation model. The construction of the image segmentation model mainly uses a bidirectional cross-frame reconstruction strategy, a coarse segmentation strategy, an edge feature fusion strategy, and a fine segmentation strategy. Figure 7c As shown, first, a bidirectional cross-frame reconstruction strategy is implemented through a forward reconstruction network and a reverse reconstruction network. The adjacent images of the target frame image (i.e., the previous frame image and the next frame image) contain a large number of similar features with the target frame image, and the visual differences between them are mainly concentrated on the edge details. Therefore, by reconstructing the target frame image based on the previous frame image and the next frame image, and by using a bidirectional reconstruction strategy to reconstruct the previous frame image and the next frame image through the target frame image, the image segmentation model can pay more attention to the section edge details, which is beneficial to improving the accuracy of the image segmentation model.

[0193] At the same time, the coarse segmentation strategy is to achieve simple segmentation of the target frame image through the initial segmentation network. Through the coarse segmentation strategy, some basic features required for image segmentation can be obtained; then, in the edge feature fusion strategy, more edge features can be introduced on the basis of the basic features required for image segmentation, so as to activate the edge area; the edge feature fusion strategy is equivalent to the residual merging module and decoding layer in the multi-feature fusion network. Finally, the fine segmentation strategy means: by performing activation mapping on the features obtained by the edge feature fusion strategy, the target image area is predicted; the fine segmentation strategy is equivalent to the activation function layer in the multi-feature fusion network.

[0194] S604: Acquire a scanning image sequence obtained by performing spatial cross-sectional layer-by-layer scanning on the designated object.

[0195] In the embodiment of the present application, the specific implementation of step S604 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0196] S605 , calling an image segmentation model to perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of any scanned image.

[0197] In the embodiment of the present application, the specific implementation of step S605 can refer to the specific implementation of step S203 in the above embodiment, which will not be repeated here.

[0198] S606: predict a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship characterizing a cross-sectional edge change from an adjacent image to a corresponding scanned image.

[0199] In the embodiment of the present application, the specific implementation of step S606 can refer to the specific implementation of steps S203 and S303 in the above embodiment, which will not be repeated here.

[0200] S607: Based on any scanned image and the inverse change relationship that characterizes the change of the cross-sectional edge from the scanned image to the corresponding adjacent image, predict an adjacent edge feature map of an adjacent image of any scanned image.

[0201] In the embodiment of the present application, the specific implementation of step S607 can refer to the specific implementation of step S204 in the above embodiment, which will not be repeated here.

[0202] S608: Determine a target image region presenting a cross section of a designated object in any scanned image according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map.

[0203] In the embodiment of the present application, the specific implementation of step S608 can refer to the specific implementation of step S204 and steps S305 to S308 in the above embodiment, which will not be repeated here.

[0204] In an embodiment of the present application, the constructed image segmentation model, based on the rough segmentation performed by the initial segmentation network, will also add edge features extracted by the forward reconstruction network and the reverse reconstruction network constructed by the bidirectional cross-frame reconstruction strategy, so that the multi-feature fusion network in the image segmentation model can use richer edge feature information to accurately distinguish the edge of the section, thereby helping to improve the accuracy of image segmentation. In addition, the embodiment of the present application can also train the entire image segmentation model by fusing the loss values ​​of the initial segmentation network, the forward reconstruction network, and the reverse reconstruction network on the basis of the loss values ​​originally determined based on the target prediction area and the training image area corresponding to each training image, which is conducive to further improving the segmentation accuracy of the image segmentation model.

[0205] In one embodiment of the present application, another image processing method is provided, which can be applied to Figure 1 The image processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 8 FIG. 4 is a flow chart showing another image processing method. Figure 2 , Figure 3 and Figure 6This paper expands on the method shown in .

[0206] Specifically, we can Figure 3 and Figure 6 The image segmentation model in is used as an initial classifier, and then multiple image segmentation models can be trained, thereby having multiple initial classifiers; finally, a total segmentation model can be formed by multiple initial classifiers. The multiple initial classifiers include multiple forward reconstruction networks, multiple reverse reconstruction networks, and multiple initial segmentation networks, which is equivalent to the total segmentation model including multiple forward change relationships, multiple reverse change relationships, and multiple initial edge feature extraction processes with different feature extraction parameters.

[0207] Among them, S801 to S803 are described in detail as follows:

[0208] S801, obtaining a scanning image sequence obtained by performing spatial cross-sectional layer-by-layer scanning on a designated object.

[0209] In the embodiment of the present application, the specific implementation of step S801 can also refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0210] S802, calling each initial classifier in the overall segmentation model, processing any scanned image in the scanned image sequence, and obtaining an initial edge feature map, a reference edge feature map, and an adjacent edge feature map outputted by each initial classifier.

[0211] In the embodiment of the present application, the specific implementation of each initial classifier processing any scanned image can refer to the specific implementation of steps S202 to S204 and steps S302 to S306, which will not be repeated here.

[0212] S803, performing integrated analysis processing on multiple initial edge feature maps, multiple reference edge feature maps, and multiple adjacent edge feature maps to obtain a target image region presenting a cross section of a specified object in any scanned image.

[0213] In the embodiment of the present application, the initial edge feature map, the reference edge feature map, and the adjacent edge feature map output by the same initial classifier have a corresponding relationship.

[0214] Then, the specific method of integrated analysis and processing can be: according to each initial edge feature map, and the reference edge feature map and adjacent edge feature map corresponding to each initial edge feature map, determine each reference image area presenting the cross-section of a specified object in any scanned image; perform regional fusion of multiple reference image areas to obtain a target image area.

[0215] Specifically, regional fusion can be performed through integrated learning algorithms such as averaging method and voting method, which are not limited here.

[0216] Optionally, the specific method of integrated analysis and processing may also be: based on each initial edge feature map, and the reference edge feature map and adjacent edge feature map corresponding to each initial edge feature map, obtain each prediction probability matrix of any scanned image; perform weighted addition processing on the elements corresponding to the same pixels in multiple prediction probability matrices, and obtain a new prediction probability matrix based on the processing results; based on the new prediction probability matrix, perform cross-sectional segmentation on any scanned image to obtain the target image area.

[0217] Wherein, the specific implementation of obtaining each prediction probability matrix can refer to step S307, which will not be described in detail here. The weights of the elements in the same prediction probability matrix can be the same, and the weights of each prediction probability matrix can be manually set at the beginning, and can be changed in the process of model training to obtain the final weights of each prediction probability matrix.

[0218] In the embodiment of the present application, based on the strategy of ensemble learning, multiple weak classifiers (i.e., initial classifiers) can be combined to obtain a better and more comprehensive strong classifier (i.e., the total segmentation model). Among them, the embodiment of the present application integrates the results of all classifiers by ensemble analysis methods such as averaging method and voting method, which can further reduce the error of the prediction result of a single classifier, thereby improving the robustness and generalization ability of the total segmentation model, and further improving the accuracy of image segmentation.

[0219] Here, an apparatus embodiment of the present application is introduced, which can be used to execute the image processing method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the image processing method of the present application.

[0220] The present application embodiment provides an image processing device, such as Fig. 9 As shown, the device includes an acquisition unit 901, a processing unit 902 and a determination unit 903, wherein:

[0221] An acquisition unit 901 is used to acquire a scanning image sequence obtained by performing spatial cross-section layer-by-layer scanning on a designated object;

[0222] The processing unit 902 is used to perform initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of the any scanned image;

[0223] The processing unit 902 is further configured to predict a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship representing a change in a cross-sectional edge from the adjacent image to the corresponding scanned image;

[0224] The processing unit 902 is further configured to predict an adjacent edge feature map of an adjacent image of the any scanned image based on the any scanned image and an inverse change relationship representing a cross-sectional edge change from the scanned image to a corresponding adjacent image;

[0225] The determination unit 903 is used to determine a target image region presenting a cross section of the designated object in any one of the scanned images according to the initial edge feature map, the reference edge feature map and the adjacent edge feature map.

[0226] In one embodiment of the present application, based on the aforementioned scheme, when the determination unit 903 determines the target image area presenting the cross section of the specified object in any one of the scanned images according to the initial edge feature map, the reference edge feature map and the adjacent edge feature map, it can be specifically used to: perform feature fusion on the reference edge feature map and the adjacent edge feature map to obtain a fused edge feature map; perform feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain a target feature map of any one of the scanned images; wherein the size of the target feature map is the same as the size of the any one of the scanned images; perform pixel classification processing according to the target feature map to obtain a prediction probability matrix of any one of the scanned images; wherein each element in the prediction probability matrix is ​​used to characterize the probability that the object presented by the corresponding pixel in the target feature map is the cross section of the specified object; based on the prediction probability matrix of any one of the scanned images, perform cross-section segmentation on any one of the scanned images to obtain the target image area.

[0227] In one embodiment of the present application, the reference edge feature map includes a reference encoding feature map and a reference decoding feature map, the adjacent edge feature map includes an adjacent encoding feature map and an adjacent decoding feature map, any decoding feature map is obtained by decoding the corresponding encoding feature map, and the fused edge feature map includes a fused encoding feature map and a fused decoding feature map; based on the aforementioned scheme, when the determination unit 903 performs feature fusion on the reference edge feature map and the adjacent edge feature map to obtain the fused edge feature map, it can be specifically used to: perform feature fusion on the reference encoding feature map and the adjacent encoding feature map to obtain the fused encoding feature map; perform feature fusion on the reference decoding feature map and the adjacent decoding feature map to obtain the fused decoding feature map.

[0228] In one embodiment of the present application, the initial edge feature map includes an initial encoding feature map and an initial decoding feature map; based on the aforementioned scheme, when the determination unit 903 performs feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain the target feature map of any scanned image, it can be specifically used to: perform residual merging processing on the fused encoding feature map and the initial encoding feature map to obtain a residual encoding feature map; perform decoding processing on the residual encoding feature map to obtain a residual decoding feature map; perform residual merging processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain the target feature map.

[0229] In one embodiment of the present application, based on the aforementioned scheme, when the determination unit 903 performs residual merging processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain the target feature map, it can be specifically used to: perform cross multiplication processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain a processed feature map; perform feature convolution processing on the processed feature map to obtain a convolved feature map; perform feature fusion on the convolved feature map and the initial encoding feature map to obtain the target feature map.

[0230] In one embodiment of the present application, the adjacent images of any scanned image include the previous frame image and the next frame image of any scanned image in the scanned image sequence; based on the aforementioned scheme, when the processing unit 902 predicts the reference edge feature map of any scanned image based on the adjacent images of any scanned image and the positive change relationship characterizing the cross-sectional edge change from the adjacent image to the corresponding scanned image, it can be specifically used to: splice the previous frame image and the next frame image of any scanned image to obtain a target spliced ​​image; obtain the reference edge feature map of any scanned image based on the target spliced ​​image and the positive change relationship.

[0231] In one embodiment of the present application, the reference edge feature map includes a reference encoding feature map and a reference decoding feature map, and the forward change relationship is mapped to network parameters in a forward reconstruction network contained in an image segmentation model; based on the aforementioned scheme, when the processing unit 902 obtains the reference edge feature map of any scanned image based on the target spliced ​​image and the forward change relationship, it can be specifically used to: call the forward reconstruction network, downsample the target spliced ​​image to obtain a reference low-dimensional feature map; perform high-dimensional convolution processing on the reference low-dimensional feature map to obtain a reference encoding feature map; and decode the reference encoding feature map to obtain the reference decoding feature map.

[0232] In one embodiment of the present application, the number of the initial edge feature maps includes multiple, and different initial edge feature maps are obtained by performing initial edge feature extraction processing on any of the scanned images based on different feature extraction parameters; the number of the reference edge feature maps includes multiple, and different reference edge feature maps are obtained based on adjacent images of any of the scanned images and multiple positive change relationship predictions; the adjacent edge feature maps include multiple, and different adjacent edge feature maps are obtained based on any of the scanned images and multiple reverse change relationship predictions; based on the aforementioned scheme, when the determination unit 903 determines the target image area presenting the cross-section of the specified object in any of the scanned images based on the initial edge feature map, the reference edge feature map and the adjacent edge feature map, it can be specifically used to: integrate and analyze the multiple initial edge feature maps, the multiple reference edge feature maps and the multiple adjacent edge feature maps to obtain the target image area.

[0233] In one embodiment of the present application, there is a one-to-one correspondence between the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps; based on the aforementioned scheme, when the determination unit 903 performs integrated analysis and processing on the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps to obtain the target image area, it can be specifically used to: determine each reference image area presenting the cross-section of the specified object in any scanned image based on each initial edge feature map, and the reference edge feature map and the adjacent edge feature map corresponding to each initial edge feature map; perform regional fusion on multiple reference image areas to obtain the target image area.

[0234] In one embodiment of the present application, there is a one-to-one correspondence between the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps; based on the aforementioned scheme, when the determination unit 903 performs integrated analysis processing on the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps to obtain the target image area, it can be specifically used to: obtain each prediction probability matrix of any scanned image based on each initial edge feature map, and the reference edge feature map and the adjacent edge feature map corresponding to each initial edge feature map; perform weighted addition processing on the elements corresponding to the same pixel in the multiple prediction probability matrices, and obtain a new prediction probability matrix according to the processing results; and perform cross-sectional segmentation on any scanned image based on the new prediction probability matrix to obtain the target image area.

[0235] In one embodiment of the present application, the target image region is obtained by processing any scanned image and any adjacent image of the scanned image through an image segmentation model; based on the above scheme, the processing unit 902 can also be used to: obtain a training image sequence obtained by performing spatial section-by-section scanning on the training object, and a training image region presenting the section of the training object in each training image included in the training image sequence; call the image segmentation model to be trained, process each training image, and obtain a target prediction region presenting the section of the training object in each training image; wherein the image segmentation model to be trained includes an initial segmentation network for initial feature extraction processing, a bidirectional reconstruction network including a forward reconstruction network and a reverse reconstruction network, and a fusion decoding network for determining the target prediction region, the forward reconstruction network is used to construct the forward change relationship by reconstructing each training image through its adjacent images, and the reverse reconstruction network is used to construct the reverse change relationship by reconstructing each training image through its adjacent images; based on the target prediction region and the training image region corresponding to each training image, the image segmentation model to be trained is trained.

[0236] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 calls the image segmentation model to be trained and processes the respective training images to obtain the target prediction area presenting the cross section of the training object in the respective training images, it can be specifically used to: call the inverse reconstruction network to predict the front frame reconstructed image and the rear frame reconstructed image corresponding to the respective training images; when the processing unit 902 trains the image segmentation model to be trained based on the target prediction area and the training image area corresponding to the respective training images, it can be specifically used to: determine a first loss value based on the front frame reconstructed image and the front frame image of the respective training images, and determine a second loss value based on the rear frame reconstructed image and the rear frame image of the respective training images; train the image segmentation model to be trained based on the first loss value, the second loss value, and the target prediction area and the training image area corresponding to the respective training images.

[0237] In one embodiment of the present application, the adjacent images of each training image include a front frame image and a rear frame image of each training image; based on the aforementioned scheme, when the processing unit 902 calls the image segmentation model to be trained and performs image segmentation processing on each training image to obtain the target prediction area presenting the training stereo object in each training image, it can be specifically used to: call the forward reconstruction network to splice the front frame image and the rear frame image of each training image to obtain the training spliced ​​image of each training image; obtain the reconstructed image of each training image according to the training spliced ​​image prediction of each training image; when the processing unit 902 trains the image segmentation model to be trained based on the target prediction area and the training image area corresponding to each training image, it can be specifically used to: determine a third loss value based on the reconstructed image of each training image and the each training image; and train the image segmentation model to be trained based on the third loss value, and the target prediction area and the training image area corresponding to each training image.

[0238] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0239] The device provided in the above embodiment can be arranged in a terminal device or in a server. The device provided in the embodiment of the present application can obtain the initial edge feature map of the scanned image through the initial edge feature extraction process, and further predict the reference edge feature map of the scanned image and the adjacent edge feature map of the adjacent image of the scanned image through the forward reconstruction network and the reverse reconstruction network. Since the cross-sectional edges between the scanned image and its adjacent images have spatial correlation, the reference edge feature map and the adjacent edge feature map can provide richer edge feature information for the subsequent segmentation of the cross section, which is equivalent to capturing more cross-sectional edge details, which is conducive to the subsequent use of more edge information for accurate image segmentation and is conducive to improving the accuracy of image segmentation.

[0240] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above image processing method.

[0241] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0242] It should be noted that Fig.10The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0243] like Fig.10 As shown, the computer system 1000 includes a processor (Central Processing Unit, CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (Random Access Memory, RAM) 1003, such as executing the method in the above embodiment. In RAM 1003, various programs and data required for system operation are also stored. CPU 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Input / output (Input / Output, I / O) interface 1005 is also connected to bus 1004.

[0244] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0245] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a processor (CPU) 1001, various functions defined in the system of the present application are executed.

[0246] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0247] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0248] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0249] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the image processing method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0250] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device performs the image processing method as described above in the above embodiments.

[0251] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0252] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0253] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a scanning image sequence obtained by scanning the designated object layer by layer in spatial cross section; Performing initial edge feature extraction processing on any scanned image in the scanned image sequence to obtain an initial edge feature map of the any scanned image; Predicting a reference edge feature map of any scanned image based on an adjacent image of any scanned image and a positive change relationship characterizing a change in a cross-sectional edge from an adjacent image to a corresponding scanned image; Based on the any scanned image and the inverse change relationship representing the change of the cross-sectional edge from the scanned image to the corresponding adjacent image, predicting an adjacent edge feature map of an adjacent image of the any scanned image; A target image region presenting a cross section of the designated object in any one of the scanned images is determined according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map.

2. The method according to claim 1, characterized in that The step of determining a target image region presenting a cross section of the designated object in any one of the scanned images according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map comprises: Performing feature fusion on the reference edge feature map and the adjacent edge feature map to obtain a fused edge feature map; Performing feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain a target feature map of any scanned image; wherein the size of the target feature map is the same as the size of any scanned image; Perform pixel classification processing according to the target feature map to obtain a prediction probability matrix of any scanned image; wherein each element in the prediction probability matrix is ​​used to represent the probability that the object presented by the corresponding pixel in the target feature map is a cross section of the specified object; Based on the prediction probability matrix of any scanned image, any scanned image is segmented to obtain the target image area.

3. The method according to claim 2, characterized in that The reference edge feature map includes a reference encoding feature map and a reference decoding feature map, the adjacent edge feature map includes an adjacent encoding feature map and an adjacent decoding feature map, any decoding feature map is obtained by decoding the corresponding encoding feature map, and the fused edge feature map includes a fused encoding feature map and a fused decoding feature map; The step of fusing the reference edge feature map and the adjacent edge feature map to obtain a fused edge feature map comprises: Performing feature fusion on the reference coding feature map and the adjacent coding feature map to obtain the fused coding feature map; The reference decoding feature map and the adjacent decoding feature map are feature-fused to obtain the fused decoding feature map.

4. The method according to claim 3, characterized in that The initial edge feature map includes an initial encoding feature map and an initial decoding feature map; the step of performing feature residual fusion processing on the fused edge feature map and the initial edge feature map to obtain a target feature map of any scanned image includes: Performing residual merging processing on the fused coding feature map and the initial coding feature map to obtain a residual coding feature map; Decoding the residual coding feature map to obtain a residual decoding feature map; The fused decoding feature map, the residual decoding feature map and the initial decoding feature map are subjected to residual merging processing to obtain the target feature map.

5. The method according to claim 4, characterized in that The performing residual merging processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain the target feature map includes: Performing cross multiplication processing on the fused decoding feature map, the residual decoding feature map and the initial decoding feature map to obtain a processed feature map; Performing feature convolution processing on the processed feature map to obtain a convolved feature map; The convolved feature map and the initial encoded feature map are subjected to feature fusion to obtain the target feature map.

6. The method according to claim 1, characterized in that The adjacent images of any scanned image include a previous frame image and a subsequent frame image of the any scanned image in the scanned image sequence; the reference edge feature map of any scanned image is predicted based on the adjacent images of the any scanned image and the positive change relationship representing the cross-sectional edge change from the adjacent image to the corresponding scanned image, including: Splicing the previous frame image and the next frame image of any scanned image to obtain a target spliced ​​image; Based on the target spliced ​​image and the positive change relationship, a reference edge feature map of any scanned image is obtained.

7. The method according to claim 6, characterized in that The reference edge feature map includes a reference encoding feature map and a reference decoding feature map, and the forward change relationship is mapped to network parameters in a forward reconstruction network included in the image segmentation model; The step of obtaining a reference edge feature map of any scanned image based on the target stitched image and the positive change relationship includes: Calling the forward reconstruction network to downsample the target spliced ​​image to obtain a reference low-dimensional feature map; Performing high-dimensional convolution processing on the reference low-dimensional feature map to obtain a reference coding feature map; The reference encoding feature map is decoded to obtain the reference decoding feature map.

8. The method according to claim 1, characterized in that The number of the initial edge feature maps includes a plurality, and different initial edge feature maps are obtained by performing initial edge feature extraction processing on any scanned image based on different feature extraction parameters; The number of the reference edge feature maps includes multiple reference edge feature maps, and different reference edge feature maps are predicted based on adjacent images of any scanned image and multiple positive change relationships; The adjacent edge feature maps include a plurality of adjacent edge feature maps, and different adjacent edge feature maps are predicted based on any one of the scanned images and a plurality of inverse change relationships; The step of determining a target image region presenting a cross section of the designated object in any one of the scanned images according to the initial edge feature map, the reference edge feature map, and the adjacent edge feature map comprises: The multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps are integrated and analyzed to obtain the target image area.

9. The method according to claim 8, characterized in that There is a one-to-one correspondence between the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps; the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps are integrated and analyzed to obtain the target image area, including: Determine, according to each initial edge feature map, and the reference edge feature map and the adjacent edge feature map corresponding to each initial edge feature map, each reference image region presenting the cross section of the designated object in any one of the scanned images; A plurality of reference image regions are regionally fused to obtain the target image region.

10. The method according to claim 8, characterized in that There is a one-to-one correspondence between the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps; the multiple initial edge feature maps, the multiple reference edge feature maps, and the multiple adjacent edge feature maps are integrated and analyzed to obtain the target image area, including: Based on each initial edge feature map, and a reference edge feature map and an adjacent edge feature map corresponding to each initial edge feature map, obtaining each prediction probability matrix of any scanned image; Performing weighted addition processing on elements corresponding to the same pixel in multiple prediction probability matrices, and obtaining a new prediction probability matrix according to the processing results; Based on the new prediction probability matrix, any scanned image is segmented to obtain the target image area.

11. The method according to any one of claims 1 to 10, characterized in that The target image area is obtained by processing any scanned image and any adjacent image of the scanned image through an image segmentation model; the method further includes: Acquire a training image sequence obtained by performing spatial cross-section layer-by-layer scanning on a training object, and a training image region presenting a cross-section of the training object in each training image included in the training image sequence; Calling the image segmentation model to be trained, processing each training image, and obtaining the target prediction region of the cross section of the training object in each training image; wherein the image segmentation model to be trained includes an initial segmentation network for initial feature extraction processing, a bidirectional reconstruction network including a forward reconstruction network and a reverse reconstruction network, and a fusion decoding network for determining the target prediction region, the forward reconstruction network is used to construct the forward change relationship by reconstructing each training image through adjacent images of each training image, and the reverse reconstruction network is used to construct the reverse change relationship by reconstructing adjacent images of each training image through each training image; The image segmentation model to be trained is trained based on the target prediction area and the training image area corresponding to each training image.

12. The method according to claim 11, characterized in that The calling of the image segmentation model to be trained, processing each of the training images, and obtaining a target prediction region presenting a cross section of the training object in each of the training images, comprises: Calling the reverse reconstruction network to predict and obtain a previous frame reconstructed image and a subsequent frame reconstructed image corresponding to each training image; The training of the image segmentation model to be trained based on the target prediction area and the training image area corresponding to each training image includes: Determine a first loss value based on a front frame reconstructed image and a front frame image of each training image, and determine a second loss value based on a rear frame reconstructed image and a rear frame image of each training image; The image segmentation model to be trained is trained based on the first loss value, the second loss value, and the target prediction area and the training image area corresponding to each training image.

13. The method according to claim 11, characterized in that The adjacent images of each training image include a front frame image and a rear frame image of each training image; calling the image segmentation model to be trained to perform image segmentation processing on each training image to obtain a target prediction area in each training image where the training stereo object is presented, includes: Calling the forward reconstruction network to splice the front frame image and the rear frame image of each training image to obtain a training spliced ​​image of each training image; Predicting a reconstructed image of each training image based on the training spliced ​​image of each training image; The training of the image segmentation model to be trained based on the target prediction area and the training image area corresponding to each training image includes: Determining a third loss value based on the reconstructed image of each training image and each training image; The image segmentation model to be trained is trained based on the third loss value, and the target prediction area and the training image area corresponding to each training image.

14. A computer storage medium, characterized in that: The computer storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 13.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method as described in any one of claims 1 to 13.

Images