Image processing method, electronic device, and storage medium

Abstract

The application discloses an image processing method, an electronic device and a storage medium. The method is applied to the field of image processing and includes the following steps: acquiring multiple images, wherein the display content of the images at least contains a monitoring area of a target part of a to-be-monitored object; performing semantic segmentation on the images to obtain a first regional feature of the monitoring area in the images; determining a second regional feature of the monitoring area based on a dependency relationship between a plurality of prototypes and the first regional feature, wherein different prototypes are used to represent different types of monitoring areas; and identifying feature information of the monitoring area based on the first regional feature and the second regional feature, and determining an identification result of the monitoring area. The application solves the technical problem of low identification accuracy of a to-be-monitored image in the related art.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of technology, image processing techniques are increasingly being applied to various aspects of life, such as medicine and education. Current image processing methods, when processing target areas in an image, simply extract features from the target area and then identify those features to obtain the recognition result. However, when the image clarity is low, the accuracy of the extracted features is also low, leading to a low accuracy rate in identifying target areas in the image.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, and a storage medium to at least solve the technical problem of low recognition accuracy in the related art when recognizing images to be monitored.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring multiple images, wherein the display content of the images at least includes a monitoring region of a target part of an object to be monitored; performing semantic segmentation on the images to obtain a first region feature of the monitoring region in the images; determining a second region feature of the monitoring region based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring regions; and identifying feature information of the monitoring region based on the first region feature and the second region feature to determine the identification result of the monitoring region.

[0006] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: responding to an input command applied to an operation interface, displaying multiple images on the operation interface, wherein the displayed content of the images at least includes a monitoring area of ​​a target part of an object to be monitored; responding to an image processing command applied to the operation interface, displaying an identification result of the monitoring area on the operation interface, wherein the identification result is obtained by identifying feature information of the monitoring area based on a first region feature and a second region feature, the second region feature being determined based on the dependency relationship between a plurality of pre-constructed prototypes and the first region feature, and the first region feature being obtained by semantic segmentation of a medical image.

[0007] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: displaying multiple images on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the displayed content of the images at least includes a monitoring area of ​​a target part of an object to be monitored; performing semantic segmentation on the images to obtain a first region feature of the monitoring area in the images; determining a second region feature of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring areas; identifying feature information of the monitoring area based on the first region feature and the second region feature to determine the identification result of the monitoring area; and driving the VR device or AR device to render and display the identification result.

[0008] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: acquiring multiple images by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being multiple images, and the display content of the images at least including a monitoring area of ​​a target part of an object to be monitored; performing semantic segmentation on the images to obtain a first region feature of the monitoring area in the images; determining a second region feature of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring areas; identifying feature information of the monitoring area based on the first region feature and the second region feature to determine the identification result of the monitoring area; and outputting the identification result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the identification result.

[0009] According to another aspect of the embodiments of this application, an image processing apparatus is also provided, comprising: an acquisition module for acquiring multiple images, wherein the display content of the images at least includes a monitoring region of a target part of an object to be monitored; a segmentation module for performing semantic segmentation on the images to obtain a first region feature of the monitoring region in the images; a first determination module for determining a second region feature of the monitoring region based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring regions; and a second determination module for identifying feature information of the monitoring region based on the first region feature and the second region feature, and determining the identification result of the monitoring region.

[0010] According to another aspect of the embodiments of this application, an image processing apparatus is also provided, comprising: a first display module, configured to display multiple images on the operation interface in response to an input command applied to an operation interface, wherein the displayed content of the images at least includes a monitoring area of ​​a target part of an object to be monitored; and a second display module, configured to display the recognition result of the monitoring area on the operation interface in response to an image processing command applied to the operation interface, wherein the recognition result is obtained by recognizing feature information of the monitoring area based on a first region feature and a second region feature, the second region feature being determined based on the dependency relationship between a plurality of pre-constructed prototypes and the first region feature, and the first region feature being obtained by semantic segmentation of a medical image.

[0011] According to another aspect of the embodiments of this application, an image processing apparatus is also provided, comprising: a display module for displaying multiple images on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the display content of the images at least includes a monitoring region of a target part of an object to be monitored; a segmentation module for performing semantic segmentation on the images to obtain a first region feature of the monitoring region in the images; a first determination module for determining a second region feature of the monitoring region based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring regions; a second determination module for identifying feature information of the monitoring region based on the first region feature and the second region feature, and determining the identification result of the monitoring region; and a driving module for driving the VR device or AR device to render and display the identification result.

[0012] According to another aspect of the embodiments of this application, an image processing apparatus is also provided, comprising: an acquisition module, configured to acquire multiple images by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being multiple images, and the display content of the images at least including a monitoring region of a target part of an object to be monitored; a segmentation module, configured to perform semantic segmentation on the images to obtain a first region feature of the monitoring region in the images; a first determination module, configured to determine a second region feature of the monitoring region based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring regions; a second determination module, configured to identify feature information of the monitoring region based on the first region feature and the second region feature, and determine the identification result of the monitoring region; and an output module, configured to output the identification result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the identification result.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the above when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located executes any of the above methods.

[0015] According to another aspect of the embodiments of this application, a method for diagnosing pulmonary nodules is also provided, comprising: acquiring multiple medical images, wherein the multiple medical images contain pulmonary nodules; performing semantic segmentation on the medical images to obtain first nodule features of the pulmonary nodules in the medical images; determining second nodule features of the pulmonary nodules based on the dependency relationship between multiple pre-constructed prototypes and the first nodule features, wherein different prototypes are used to characterize different types of pulmonary nodules; diagnosing the pulmonary nodules based on the first nodule features and the second nodule features to obtain a diagnostic result of the pulmonary nodules, wherein the diagnostic result is used to characterize whether the pulmonary nodules are benign or malignant nodules.

[0016] According to another aspect of the embodiments of this application, a lung nodule diagnostic device is also provided, comprising: an acquisition module for acquiring multiple medical images, wherein the multiple medical images contain lung nodules; a segmentation module for performing semantic segmentation on the medical images to obtain first nodule features of the lung nodules in the medical images; a determination module for determining second nodule features of the lung nodules based on the dependency relationship between multiple pre-constructed prototypes and the first nodule features, wherein different prototypes are used to characterize different types of lung nodules; and a diagnosis module for diagnosing the lung nodules based on the first nodule features and the second nodule features to obtain a diagnostic result of the lung nodules, wherein the diagnostic result is used to characterize whether the lung nodules are benign or malignant nodules.

[0017] In this embodiment, the method involves acquiring multiple images; performing semantic segmentation on the images to obtain first region features of the monitored area; determining second region features of the monitored area based on the dependency relationship between multiple pre-constructed prototypes and the first region features; and identifying the feature information of the monitored area based on the first and second region features to determine the identification result of the monitored area. It is noteworthy that this application not only performs semantic segmentation on the images to obtain region features but also combines the dependency relationship between prototypes and region features to ensure that the final region features better match the attributes of the target part itself, resulting in higher feature extraction accuracy. This achieves the goal of more accurately identifying the target part of the monitored object, thereby improving the technical effect of increasing the identification accuracy of the target part of the monitored object and solving the technical problem of low identification accuracy in related technologies for identifying monitored images.

[0018] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the image processing method of this application;

[0021] Figure 2 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of an image processing method according to Embodiment 1 of this application;

[0023] Figure 4 This is a schematic diagram of an optional image processing method according to Embodiment 1 of this application.

[0024] Figure 5 This is a flowchart of an image processing method according to Embodiment 2 of this application;

[0025] Figure 6 This is a schematic diagram of an optional user interface according to Embodiment 2 of this application;

[0026] Figure 7 This is a flowchart of an image processing method according to Embodiment 3 of this application;

[0027] Figure 8 This is a flowchart of an image processing method according to Embodiment 4 of this application;

[0028] Figure 9 This is a flowchart of a lung nodule diagnosis method according to Embodiment 5 of this application.

[0029] Figure 10 This is a schematic diagram illustrating an optional comparison between reader research and artificial intelligence according to Embodiment 5 of this application;

[0030] Figure 11 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 6 of this application;

[0031] Figure 12 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 7 of this application;

[0032] Figure 13 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 8 of this application;

[0033] Figure 14This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 9 of this application;

[0034] Figure 15 This is a schematic diagram of the structure of a lung nodule diagnostic device according to Embodiment 10 of this application;

[0035] Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0039] U-shaped neural network: It includes a feature extraction network (encoder) for feature extraction to obtain abstract semantic features, and a feature fusion network (decoder) to restore the original image size using the previously encoded abstract features, and finally obtain the segmentation result (mask image). The feature extraction network and the feature fusion network are connected to obtain a U-shaped neural network.

[0040] Self-attention models: Attention models where queries, keys, and values ​​come from the same set of inputs, and can better understand contextual information when processing sequences.

[0041] Cross-attention model: An attention model where the key and value are the same but different from the query.

[0042] Prototype: Images with similar features are clustered in the representation space using a clustering algorithm, and the resulting cluster centers are used as the prototypes of that category.

[0043] Example 1

[0044] According to an embodiment of this application, an image processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the image processing method of this application. Figure 1 As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 via a network. The virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The terminal 104 is not limited to PCs, mobile phones, tablets, etc. The server 102 can be a server corresponding to a media file operator. The network includes, but is not limited to: wide area network, metropolitan area network, or local area network.

[0046] Optionally, the virtual reality device 104 in this embodiment includes a memory, a processor, and a transmission device. The memory stores an application program that can perform the following actions: acquiring multiple images, wherein the displayed content of the images at least includes a monitoring area of ​​the target part of the object to be monitored; performing semantic segmentation on the images to obtain a first region feature of the monitoring area in the images; determining a second region feature of the monitoring area based on the dependency relationship between multiple pre-built prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring areas; identifying the feature information of the monitoring area based on the first region feature and the second region feature, and determining the recognition result of the monitoring area. This solves the technical problem of low recognition accuracy in related technologies for recognizing images to be monitored, and achieves the goal of accurately recognizing the target part of the object to be monitored.

[0047] The terminal in this embodiment can be used to display multiple images on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; perform semantic segmentation on the images to obtain the first region features of the monitoring area in the images; determine the second region features of the monitoring area based on the dependency relationship between multiple pre-built prototypes and the first region features, wherein different prototypes are used to represent different types of monitoring areas; identify the feature information of the monitoring area based on the first region features and the second region features to determine the identification result of the monitoring area; and drive the VR device or AR device to render and display the identification result.

[0048] Optionally, the eye-tracking HMD (Head-Mounted Display) and eye-tracking module in the virtual reality device 104 of this embodiment function the same as in the embodiments described above. That is, the screen in the HMD is used to display real-time images, and the eye-tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The terminal in this embodiment acquires the user's position and movement information in real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in virtual three-dimensional space, as well as the user's field of vision orientation in virtual three-dimensional space.

[0049] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The AR / VR device (or mobile device) shown is an example of a computing node in computing environment 201. Figure 2 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (represented as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0050] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0051] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0052] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers within a Pod handle requests related to one or more corresponding functions of the service. Proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0053] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201, and executing one or more functions of one service may require calling one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0054] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0055] Under the aforementioned operating environment, this application provides the following: Figure 3 The image processing method shown is illustrated. It should be noted that the image processing method in this embodiment can be derived from... Figure 1 The mobile terminal in the illustrated embodiment is executed. Figure 3 This is a flowchart of an image processing method according to Embodiment 1 of this application. Figure 3 As shown, the method may include the following steps:

[0056] Step S302: Acquire multiple images, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored.

[0057] The objects to be monitored mentioned above can be parts of the human body, but are not limited to these; they can also be parts of buildings, etc. The target parts mentioned above can be specific areas of the object to be monitored that require monitoring. For example, when the object to be monitored is the lungs of a human body, the target parts could be nodules, blood vessels, trachea, etc., but are not limited to these. When the object to be monitored is a window on a building wall, the target parts could be window handles, corners of the frame, the middle part, etc., but are not limited to these. The monitoring area mentioned above can be the area containing the target parts of the object to be monitored, and can be called the Region of Interest (ROI).

[0058] In one optional embodiment, when monitoring a target area of ​​an object, firstly, an original image containing the target area of ​​the object can be acquired. Secondly, the original image can be cropped based on the monitoring area to obtain an image of the target area, i.e., the aforementioned multiple images, wherein at least one of the multiple images contains the target area of ​​the object to be monitored. For example, when monitoring nodules in the lungs, firstly, a CT image of the lungs can be acquired, where the CT image is a 3D image. Secondly, the CT image can be cropped based on the lungs to obtain a 3D cropped image. Then, in order to extract features from the 3D cropped image, the 3D cropped image can be converted into multiple 2D images (i.e., multiple images). As another example, when monitoring windows on a building wall, firstly, multiple original images of the windows can be acquired. Secondly, the multiple original images can be cropped based on the windows to obtain the aforementioned multiple images.

[0059] In another optional embodiment, when monitoring a target area of ​​an object, firstly, a video of the object being monitored can be acquired. Secondly, multiple original images can be extracted from the video. Then, the multiple original images can be cropped based on the monitoring area to obtain multiple images of the target area, i.e., the aforementioned multiple images. Each of these multiple images contains at least one target area of ​​the object being monitored. For example, when monitoring a window on a building wall, firstly, a video of the window can be acquired. Secondly, multiple original images can be extracted from the video. Then, the multiple original images can be cropped based on the window to obtain the aforementioned multiple images.

[0060] Step S304: Perform semantic segmentation on the image to obtain the first region feature of the monitored area in the image.

[0061] In one optional embodiment, after acquiring the image, semantic segmentation can be performed on the monitoring area image using a semantic segmentation model to obtain the first region features of the monitoring area. For example, contextual semantic segmentation can be performed on the monitoring area image using a semantic segmentation model to obtain the first region features of the monitoring area. Alternatively, firstly, contextual semantic segmentation can be performed on the monitoring area image using a semantic segmentation model to obtain preset region features of the monitoring area; secondly, contextual parsing can be performed on the preset region features using a semantic segmentation model to obtain the first region features of the monitoring area, but this is not limited to these methods.

[0062] It should be noted that the semantic segmentation model mentioned above can be any one or more models in related technologies that can perform semantic segmentation on the image of the monitoring area to obtain the features of the first region. No specific limitation is made in this embodiment.

[0063] Step S306: Based on the dependencies between multiple pre-built prototypes and the first region features, determine the second region features of the monitoring region, wherein different prototypes are used to characterize different types of monitoring regions.

[0064] The prototypes mentioned above can be monitoring areas of already confirmed types, and different prototypes correspond to different types. The types mentioned above can be the types of target parts. For example, when the target part included in the monitoring area is a human joint, the type of the monitoring area can be joint; when the target part included in the monitoring area is a window of a building, the type of the monitoring area can be window, but it is not limited to these.

[0065] In one optional embodiment, firstly, a dependency relationship between different prototypes and the first region feature can be constructed. Secondly, after obtaining the first region feature, the prototype corresponding to the first region feature can be determined based on the dependency relationship. Then, the first region feature and the prototype corresponding to the first region feature can be processed to obtain the second region feature of the monitoring region.

[0066] Step S308: Based on the features of the first region and the features of the second region, identify the feature information of the monitoring area and determine the identification result of the monitoring area.

[0067] The identification results mentioned above can be obtained after identifying the target parts in the monitoring area. For example, when the target part in the monitoring area is a joint, the identification result can be that the joint is in good condition or that the joint is in poor condition. When the target part in the monitoring area is a window, the identification result can be that the window meets the requirements or that the window does not meet the requirements, but it is not limited to these.

[0068] In one optional embodiment, after obtaining the first region features and the second region features, the feature information of the monitoring area can be identified based on the first region features and the second region features to obtain the identification result of the monitoring area. For example, the feature information of the monitoring area can be identified based on the first region features and the second region features respectively to obtain a first identification result and a second identification result. Then, the first identification result and the second identification result are compared, and the identification result with higher accuracy is selected as the final comparison result. Another example is that the feature information of the monitoring area can be identified based on the first region features and the second region features respectively to obtain a first identification result and a second identification result. Then, the average of the first identification result and the second identification result is taken to obtain the final identification result. Yet another example is that the first region features and the second region features can be fused first, and then the feature information of the monitoring area can be identified based on the fused region features to obtain the identification result of the monitoring area, but this is not limited to these methods.

[0069] For example, when monitoring nodules in a human lung, a CT image containing the lung can be acquired first. Then, the CT image can be cropped based on the lung region to obtain a 3D cropped image. To extract features from the 3D cropped image, it can be converted into multiple 2D images (i.e., multiple images), each containing lung nodules. A semantic segmentation model is used to perform contextual semantic segmentation on the lung image, obtaining preset region features. The semantic segmentation model can then perform contextual parsing on these preset region features to obtain the first region features of the lung image. Based on pre-built dependencies, the prototype corresponding to the first region features can be determined, and the first region features and their corresponding prototypes are processed to obtain the second region features. Finally, the feature information of the lung image can be identified based on the first and second region features, yielding a first identification result and a second identification result. The average of the first and second identification results is then obtained to obtain the final identification result.

[0070] In this embodiment, the method involves acquiring multiple images; performing semantic segmentation on the images to obtain first region features of the monitored area; determining second region features of the monitored area based on the dependency relationship between multiple pre-constructed prototypes and the first region features; and identifying the feature information of the monitored area based on the first and second region features to determine the identification result of the monitored area. It is noteworthy that this application not only performs semantic segmentation on the images to obtain region features but also combines the dependency relationship between prototypes and region features to ensure that the final region features better match the attributes of the target part itself, resulting in higher feature extraction accuracy. This achieves the goal of more accurately identifying the target part of the monitored object, thereby improving the technical effect of increasing the identification accuracy of the target part of the monitored object and solving the technical problem of low identification accuracy in related technologies for identifying monitored images.

[0071] In the above embodiments of this application, semantic segmentation of an image is performed to obtain a first region feature of the monitored area in the image, including: performing semantic segmentation of the image to obtain a semantic segmentation result and global features of the image; performing feature fusion on the semantic segmentation result and the image to obtain fused features; and performing attention processing on the global features and fused features to obtain the first region feature.

[0072] The semantic segmentation result described above can be a semantic mask M, which indicates whether a pixel in the image belongs to the target region. When a pixel belongs to the target region, the semantic segmentation result for that pixel can be 1, but it is not limited to this; it can also be 0. When the monitored object is the human lung, the semantic segmentation result M contains different voxels (e.g., nodules, blood vessels, etc.), belonging to the set {0: background, 1: lung, 2: nodule, 3: blood vessel, 4: trachea}.

[0073] In one optional embodiment, the image can first be semantically segmented by a semantic segmentation module to obtain the semantic segmentation result and global features of the image. Secondly, the semantic segmentation result and the image can be divided into small blocks based on the target region. Then, the semantic segmentation result and the image divided into small blocks can be fused for features. That is, the semantic segmentation result and the image after being divided into small blocks containing the same target region can be fused for features to obtain fused features. Finally, attention processing can be performed on the global features and the fused features to obtain the first region features.

[0074] In the above embodiments of this application, semantic segmentation of an image is performed to obtain the semantic segmentation result and global features of the image, including: extracting features from the image using the encoder module of a U-shaped neural network model to obtain the first image features of the image; extracting global features from the bottleneck layer of the U-shaped neural network model; and decoding the first image features using the encoder module of the U-shaped neural network model to obtain the semantic segmentation result.

[0075] In one optional embodiment, the first image features can be extracted by the encoder module of the U-shaped neural network model. Then, the first image features can be decoded by the decoder module of the U-shaped neural network model to obtain the semantic segmentation result. In addition, the global features of the image can be extracted by the bottleneck layer of the U-shaped neural network model, wherein the bottleneck layer is located in the middle layer of the U-shaped neural network model.

[0076] In the above embodiments of this application, feature fusion of semantic segmentation results and images is performed to obtain fused features, including: segmenting semantic segmentation results and images respectively to obtain multiple sub-segmentation results and multiple sub-images; extracting features from multiple sub-segmentation results and multiple sub-images respectively to obtain sub-segmentation features of multiple sub-segmentation results and sub-image features of multiple sub-images; and fusing sub-segmentation features and sub-image features to obtain fused features.

[0077] In an optional embodiment, after obtaining the semantic segmentation result of the image, the semantic segmentation result and the image can first be segmented based on the target region to obtain multiple sub-segmentation results and multiple sub-images. The multiple sub-segmentation results and the corresponding multiple sub-images contain the same target region. Secondly, features can be extracted from the multiple sub-segmentation results and the multiple sub-images to obtain sub-segmentation features of the multiple sub-segmentation results and sub-image features of the multiple sub-images. Then, the sub-segmentation features and sub-image features can be fused to obtain fused features.

[0078] In the above embodiments of this application, attention processing is performed on global features and fused features to obtain first region features, including: concatenating global features and fused features to obtain first concatenated features; and performing self-attention processing on the first concatenated features using a self-attention model to obtain first region features.

[0079] In one optional embodiment, the global features and fused features can first be spliced ​​together by inserting fragment positions, resulting in the first spliced ​​feature token being [q; t1, ..., t]. g ]∈R (g+1)D , where q is the global feature, t is the fused feature, R is a set of real numbers of dimension (g+1)D, where D represents the embedding dimension and g represents the number of fused features.

[0080] In another alternative embodiment, the semantic segmentation result is first divided into small image patches, which are then concatenated with the corresponding regions of the original image. Next, a sequence is generated through image patch encoding and positional encoding. Simultaneously, high-level semantic features are extracted from the convolutional neural network as global features for the nodules.

[0081] In another optional embodiment, after obtaining the first stitched feature, the first stitched feature can be processed by a self-attention model. For example, the first stitched feature can be processed by a normalization function (Norm), self-attention modeling (Service Component Architecture, SCA), and feedforward neural network (FFN) in the self-attention model, so as to obtain the first region feature.

[0082] In the above embodiments of this application, the second region feature of the monitoring region is determined based on the dependency relationship between multiple pre-built prototypes and the first region feature, including: using a cross-attention model to perform attention processing on the first region feature and multiple prototypes to obtain the second region feature.

[0083] In an optional embodiment, attention processing of the first region features and multiple prototypes can be performed using Norm, Cross Prototype Attention Module (CPA), and FFN in the cross-attention model to obtain the second region features.

[0084] In the above embodiments of this application, the method further includes: obtaining global features of different monitoring areas; clustering the global features of different monitoring areas to obtain multiple feature sets; and constructing multiple prototypes based on the central features of the multiple feature sets.

[0085] In one optional embodiment, firstly, global features of different monitoring areas can be obtained; secondly, the global features can be clustered to obtain multiple feature sets {C1, ..., C...}. N}, where C represents the features after clustering and N represents the number of features.

[0086] In another alternative embodiment, the objective function can be minimized. Multiple prototypes are obtained from the central features of multiple feature sets. Where d is the Euclidean function and p represents the global feature. It should be noted that the first prototype can be represented as P. B ∈R N / 2×D The second prototype can be represented as P M ∈R N / 2×D .

[0087] In the above embodiments of this application, after determining the second region feature of the monitoring region based on the dependency relationship between multiple pre-built prototypes and the first region feature, the method further includes: determining the target prototype that successfully matches the second region feature from the multiple pre-built prototypes; performing momentum update on the target prototype to obtain the updated region feature; and updating the multiple prototypes based on the updated region feature.

[0088] In an alternative embodiment, the momentum of the target prototype can be updated using the following formula:

[0089]

[0090] in, This is the first prototype after momentum update. The second prototype is the one after momentum update, where λ is the momentum factor, which is usually set to 0.95, but is not limited to this. Momentum update can help accelerate convergence and improve generalization ability.

[0091] In another alternative embodiment, once the updated target prototype is obtained, the updated regional features can be obtained based on the updated target prototype, and then multiple prototypes can be updated based on the updated regional features.

[0092] In the above embodiments of this application, the identification result of the monitoring area is determined by identifying the feature information of the monitoring area based on the features of the first region and the features of the second region, including: identifying the feature information of the monitoring area based on global features to obtain a first sub-identification result; identifying the feature information of the monitoring area based on the features of the first region to obtain a second sub-identification result; identifying the feature information of the monitoring area based on the features of the second region to obtain a third sub-identification result; and summarizing the first sub-identification result, the second sub-identification result, and the third sub-identification result to obtain the identification result.

[0093] In one optional embodiment, a multi-layer perceptron (MLP) can be used to identify the feature information of the detection region based on global features, first region features, and second region features, respectively, to obtain a first identification result, a second identification result, and a third identification result. Finally, the first identification result, the second identification result, and the third identification result can be summarized to obtain the identification result. For example, the average value of the first identification result, the second identification result, and the third identification result can be obtained as the identification result, or the more accurate identification result among the first identification result, the second identification result, and the third identification result can be obtained as the identification result, but it is not limited to this.

[0094] Figure 4 This is a schematic diagram of an optional image processing method according to Embodiment 1 of this application, as shown below. Figure 4 As shown, firstly, multiple images are input into a U-shaped neural network model. The U-shaped neural network can decode and encode the multiple images to obtain semantic segmentation results. Simultaneously, the U-shaped neural network model can output global features of multiple images through a bottleneck layer. Secondly, the semantic segmentation results and multiple images can be segmented to obtain multiple sub-segmentation results and multiple sub-images. Features are then extracted from these sub-segmentation results and sub-images to obtain multiple sub-segmentation features and sub-image features. Finally, these sub-segmentation features and sub-image features can be fused to obtain fused features, such as... Figure 4 The small white squares in the image can then be used to concatenate global features and fused features (i.e., block position embedding) to obtain the first concatenated feature, such as... Figure 4The system first identifies white rectangular blocks and diagonally shaded rectangular blocks. The first concatenated feature is then input into a self-attention model, passing through a normalization function, self-attention modeling, and a feedforward neural network to obtain the first region feature. This first region feature is then input into a cross-attention model, passing through a normalization function, cross-prototype attention, and a feedforward neural network to obtain the second region feature. Finally, an MLP is used to map the representation spaces of the global features, the first region feature, and the second region feature to the category space, resulting in the recognition results of three MLPs. The average of these three MLP recognition results is then obtained to obtain the final recognition result. In the self-attention model, the query, key, and value are the same, coming from the same set of inputs. However, in the cross-attention model, the query differs from the key and value.

[0095] It should be noted that the parallelogram in the figure contains multiple feature sets. Among them, the polygons connected by the arrows are prototypes. After determining the prototypes, we can determine the target prototypes that successfully match the features of the second region. Then, we can perform momentum updates on the target prototypes to obtain the updated region features. Based on the updated region features, we can update multiple prototypes.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0097] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0099] Example 2

[0100] According to an embodiment of this application, an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0101] Figure 5 This is a flowchart of an image processing method according to Embodiment 2 of this application. Figure 5 As shown, the method may include the following steps:

[0102] Step S502: In response to the input command applied to the operation interface, multiple images are displayed on the operation interface, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored.

[0103] Step S504: In response to the image processing command applied to the operation interface, the recognition result of the monitoring area is displayed on the operation interface. The recognition result is obtained by recognizing the feature information of the monitoring area based on the first region feature and the second region feature. The second region feature is determined based on the dependency relationship between multiple pre-constructed prototypes and the first region feature. The first region feature is obtained by semantic segmentation of the medical image.

[0104] Figure 6 This is a schematic diagram of an optional user interface according to Embodiment 2 of this application, such as... Figure 6As shown, the operation interface includes an input command input area, a processing command input area, and a display area. When it is necessary to monitor the target part of the object to be monitored in the image, the user can first input a display command in the input command input area of ​​the operation interface, and then the operation interface can display multiple images in the display area. Secondly, the user can input a processing command in the processing command input area, and the operation interface can display the recognition result of the monitoring area in the display area. The recognition result is obtained by recognizing the feature information of the monitoring area based on the first region feature and the second region feature. The second region feature is determined based on the dependency relationship between multiple pre-constructed prototypes and the first region feature. The first region feature is obtained by semantic segmentation of the medical image.

[0105] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0106] Example 3

[0107] According to the embodiments of this application, an image processing method applicable to virtual reality scenarios such as virtual reality (VR) devices and augmented reality (AR) devices is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0108] Figure 7 This is a flowchart of an image processing method according to Embodiment 3 of this application. Figure 7 As shown, the method may include the following steps:

[0109] Step S702: Display multiple images on the display screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the content of the displayed images at least includes the monitoring area of ​​the target part of the object to be monitored;

[0110] Step S704: Perform semantic segmentation on the image to obtain the first region features of the monitored area in the image;

[0111] Step S706: Based on the dependencies between multiple pre-built prototypes and the first region features, determine the second region features of the monitoring region, wherein different prototypes are used to characterize different types of monitoring regions;

[0112] Step S708: Identify the feature information of the monitoring area based on the features of the first region and the features of the second region, and determine the identification result of the monitoring area;

[0113] Step S7010: Drive the VR device or AR device to render and display the recognition results.

[0114] In one optional embodiment, when it is necessary to monitor a target part of an object, firstly, multiple images can be displayed on the screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object; secondly, semantic segmentation can be performed on the images to obtain the first region features of the monitoring area in the images; then, based on the dependency relationship between multiple pre-built prototypes and the first region features, the second region features of the monitoring area can be determined, wherein different prototypes are used to characterize different types of monitoring areas; then, based on the first region features and the second region features, the feature information of the monitoring area can be identified to determine the identification result of the monitoring area; finally, the VR device or AR device can be driven to render and display the identification result.

[0115] Optionally, in this embodiment, the image processing method described above can be applied to a hardware environment consisting of a server and a virtual reality device. The recognition results are displayed on the screen of the virtual reality (VR) device or augmented reality (AR) device. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The aforementioned virtual reality device is not limited to, for example, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0116] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory stores an application that can be used to perform: displaying multiple images on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; performing semantic segmentation on the images to obtain first region features of the monitoring area in the images; determining second region features of the monitoring area based on the dependencies between multiple pre-built prototypes and the first region features, wherein different prototypes are used to characterize different types of monitoring areas; identifying feature information of the monitoring area based on the first and second region features to determine the recognition result of the monitoring area; and driving the VR or AR device to render and display the recognition result.

[0117] It should be noted that the image processing method described above in this embodiment, when applied to VR or AR devices, may include... Figure 3 The method of the illustrated embodiment is used to drive VR or AR devices to display recognition results.

[0118] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to execute the above steps. The transmission device can receive media files sent by the server via a network, and can also be used for data transmission between the processor and the memory.

[0119] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0120] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be a server corresponding to a media file operator, and the network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0121] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0122] Example 4

[0123] According to an embodiment of this application, an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0124] Figure 8 This is a flowchart of an image processing method according to Embodiment 4 of this application. Figure 8 As shown, the method may include the following steps:

[0125] Step S802: Obtain multiple images by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is multiple images, and the display content of the images at least includes the monitoring area of ​​the target part of the object to be monitored.

[0126] Step S804: Perform semantic segmentation on the image to obtain the first region features of the monitored area in the image;

[0127] Step S806: Based on the dependencies between multiple pre-built prototypes and the first region features, determine the second region features of the monitoring region, wherein different prototypes are used to characterize different types of monitoring regions;

[0128] Step S808: Identify the feature information of the monitoring area based on the features of the first region and the features of the second region, and determine the identification result of the monitoring area;

[0129] Step S8010: Output the recognition result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the recognition result.

[0130] The first interface mentioned above can be an interface for obtaining multiple images from the server, and the second interface mentioned above can be an interface for sending recognition results to the server.

[0131] In one optional embodiment, when it is necessary to monitor the target part of the object to be monitored, firstly, multiple images can be obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which is multiple images, and the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; secondly, semantic segmentation can be performed on the images to obtain the first region features of the monitoring area in the images; then, based on the dependency relationship between multiple pre-constructed prototypes and the first region features, the second region features of the monitoring area can be determined, wherein different prototypes are used to represent different types of monitoring areas; then, based on the first region features and the second region features, the feature information of the monitoring area can be identified to determine the identification result of the monitoring area; finally, the identification result can be output by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which is the identification result.

[0132] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0133] Example 5

[0134] According to an embodiment of this application, a method for diagnosing pulmonary nodules is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0135] Figure 9 This is a flowchart of a method for diagnosing pulmonary nodules according to Embodiment 5 of this application. Figure 9 As shown, the method may include the following steps:

[0136] Step S902: Acquire multiple medical images, wherein the multiple medical images contain lung nodules.

[0137] The aforementioned multiple medical images can be multiple 2D medical images, which are obtained by cropping and converting the ROI region (e.g., the lung region) of a human body's computer tomography (CT) image. Among these multiple medical images, lung nodules are included.

[0138] In one alternative embodiment, when lung cancer screening is required, a medical image containing lung nodules can first be acquired, then the medical image can be cropped based on the lung region to obtain a lung image, and finally the lung image can be converted into multiple 2D medical images, wherein the multiple medical images contain lung nodules.

[0139] Step S904: Perform semantic segmentation on the medical image to obtain the first nodule feature of the lung nodules in the medical image;

[0140] In one optional embodiment, after acquiring the medical image, the lung image can first be semantically segmented using a semantic segmentation model to obtain the first nodule feature of the lung nodules. For example, the lung image can be semantically segmented using a semantic segmentation model to obtain the first nodule feature of the lung nodules in the lung image. Alternatively, the lung image can first be semantically segmented using a semantic segmentation model to obtain the first preset nodule feature of the lung nodules in the lung image, and then the first preset nodule feature can be semantically segmented using a semantic segmentation model to obtain the first nodule feature of the lung nodules in the lung image, but this is not limited to these methods.

[0141] For example, in the contextual segmentation stage, nodule contextual information has a significant impact on benign and malignant diagnosis. For instance, nodules associated with blood vessels are more likely to be malignant than isolated nodules. Therefore, a U-shaped neural network (UNet) can be used to parse the semantic mask m (i.e., the contextual semantic segmentation image) of the input image (i.e., the medical image), where the input image is a cropped image of the lung nodule ROI region from the original CT image, and the input is a three-dimensional volume composed of multiple 2D image slices. This allows for subsequent contextual modeling of both the nodule and its surrounding structures. Specifically, each voxel of m belongs to {0: background, 1: lung, 2: nodule, 3: blood vessel, 4: trachea}. This segmentation process can collect comprehensive contextual information crucial for accurate diagnosis. For diagnostic purposes, global features can be extracted from the bottleneck of the UNet as a nodule embedding q, which will be used in later diagnostic stages.

[0142] It should be noted that during the context segmentation stage, the contextual information required for differentiating between benign and malignant lung nodules includes normal lung tissue, nodules, blood vessels, and trachea. This information reflects not only the shape, location, and size of the nodule itself, but also the structural relationship between the nodule and surrounding tissues. This application uses a U-shaped convolutional neural network to perform pixel-level recognition of the aforementioned contextual semantic information, ultimately obtaining a contextual semantic segmentation map for each nodule.

[0143] Step S906: Based on the dependencies between multiple pre-built prototypes and the first nodule features, determine the second nodule features of the lung nodule, wherein different prototypes are used to characterize different types of lung nodules.

[0144] In one possible embodiment, firstly, a dependency relationship between different prototypes and the first nodule feature can be constructed. Secondly, after obtaining the first nodule feature, the prototype corresponding to the first nodule feature can be determined based on the dependency relationship. Then, the first nodule feature and the prototype corresponding to the first nodule feature can be processed to obtain the second nodule feature of the lung nodule in the lung image.

[0145] For example, in the nodule internal context parsing stage, this application designs an attention-based context parsing module to deeply analyze the nodule, integrate its contextual information, and improve the ability to distinguish between benign and malignant nodules. Specifically, for different nodules, their contextual semantic segmentation images are divided into small image blocks, which are then concatenated with the corresponding regions of the original image (i.e., the input image) to obtain multiple overlapping blocks. A string of contextual features (tokens) is generated through image block encoding and position encoding. Simultaneously, high-level semantic features are extracted from the convolutional neural network as a global representation of the nodule, also known as the nodule token. By designing a contextual self-attention module, the long-distance dependency between the nodule token and the context token is modeled, extracting relevant criteria for benign and malignant nodule identification from the contextual information. The nodule token output by the self-attention module serves as a new representation for nodule benign and malignant nodule identification.

[0146] It should be noted that during the nodule internal context resolution stage, the discriminative representation of the nodules can be enhanced by aggregating the context information generated by the segmentation model. Specifically, this can be achieved by symbolizing the context mask into a set of sequences through overlapping block embedding. The input image is also segmented into small blocks and embedded in context tokens to preserve the original image information. Furthermore, positional encoding is added in a learnable manner to preserve positional information. The nodule embedding tokens can be pre-appended to the context sequence, represented as [q; t1, ..., t...]. g ]∈R (g+1)DHere, g is the number of context tokens, and D represents the embedding dimension. These tokens can then be simultaneously modeled using self-attention, a process called SCA, to aggregate contextual information into the nodule embeddings. The nodule embedding token at the output of the last SCA block is used as the updated nodule representation. Explicitly modeling the dependencies between nodule embeddings and their background structure leads to the evolution of more discriminative representations, thereby improving the distinction between benign and malignant nodules.

[0147] For example, in the nodule prototype recall learning stage, this application designs a nodule diagnostic knowledge prototype review module. First, a nodule prototype is defined as a representative nodule with similar features. The learned lung nodules are clustered in the representation space using a clustering algorithm, and the resulting cluster centers serve as the prototypes for that category. The prototypes distinguish between benign and malignant nodules; benign prototypes are calculated from benign nodules with similar features, while malignant prototypes are derived from malignant nodules. To utilize this prototype knowledge, this application designs a cross-prototype attention module to construct the relationship between the current nodule and other prototypes. Here, the query is a representation from the current nodule, and the key and value are representations from the prototypes, respectively. The query token output by this cross-attention module serves as the final benign / malignant identification representation.

[0148] It's important to note that to retain previously acquired knowledge, a more efficient method is needed, rather than storing all learned nodes in memory, which would lead to a waste of storage and computational resources. To simplify this process, relevant nodes can be aggregated into prototype forms. For a set of nodes (i.e., multiple nodule images), they can be aggregated into N groups {C1, ..., C...}. N}, by minimizing the objective function Where d is the Euclidean distance function, p represents the nodule embedding, and the center of each cluster is used as the prototype. Considering the differences between benign and malignant nodules, the prototypes can be divided into benign and malignant groups, as determined by P. B ∈R N / 2×D and P M ∈R N / 2×DThe model, in addition to parsing the internal context, also captures the inter-layer dependencies between nodules and external prototypes. This allows PARE to explore relevant identification bases beyond individual nodules. To achieve this, this application designs a Cross-Prototype Attention (CPA) module, which utilizes nodule embeddings as queries and prototypes as keywords and values. This allows nodule embeddings to selectively engage with the most relevant parts of the prototype sequence. The state of the query at the output of the final CPA module serves as the final nodule representation to predict its malignancy label, "benign" (y=0) or "malignant" (y=1). PARE is a model for diagnosing lung nodules proposed in this application, comprising three parts: context segmentation, nodule internal context parsing, and nodule prototype recall learning.

[0149] It should be noted that this application can also update the prototype online, allowing the prototype to quickly adjust to changes in nodule embedding. For a nodule embedding q with data (x, y), its nearest prototype is selected, and then updated according to the following momentum rule:

[0150]

[0151] in, This is a benign prototype after momentum update. The malignant prototype after momentum update is λ, which is the momentum factor, usually set to 0.95, but not limited to this. Momentum update can help accelerate convergence and improve generalization ability.

[0152] Step S908: Diagnose the lung nodule based on the first nodule feature and the second nodule feature to obtain the diagnosis result of the lung nodule, wherein the diagnosis result is used to characterize whether the lung nodule is a benign or malignant nodule.

[0153] In one optional embodiment, after obtaining the first nodule features and the second nodule features, the lung nodule can be diagnosed based on the first and second nodule features to obtain a diagnostic result. For example, the lung nodule can be diagnosed based on the first and second nodule features respectively to obtain a first diagnostic result and a second diagnostic result, and then the first and second diagnostic results are compared, with the diagnostic result with higher accuracy selected as the final diagnostic result. Another example is that the lung nodule can be diagnosed based on the first and second nodule features respectively to obtain a first diagnostic result and a second diagnostic result, and then the average of the first and second diagnostic results is taken to obtain the final diagnostic result. Yet another example is that the first and second nodule features can be fused first, and then the lung nodule can be diagnosed based on the fused nodule features to obtain a diagnostic result, but this is not limited to these methods.

[0154] It should be noted that this application designs a deep supervised training mode to improve the ability to distinguish between benign and malignant nodules. The deep supervised signals are applied to the global nodal representation output by the convolutional neural network, the nodal representation output by the contextual attention module, and the nodal representation output by the cross-prototype attention module. By adding a multi-layer perceptron (MLP), the nodal representations from their respective representation spaces are mapped to the two major category spaces of benign and malignant nodules. In the inference scenario, the benign and malignant category probabilities obtained from the three MLPs are integrated into the final discrimination probability through averaging.

[0155] This application proposes a radiologist-incentivized method that simulates the diagnostic process of a radiologist, consisting of a context parsing module and a prototype review module. The context parsing module first segments the contextual structure of the nodule and then aggregates the contextual information to achieve a more comprehensive understanding of the nodule. The prototype review module utilizes prototype-based learning to compress previously learned information into a prototype for comparative analysis, which is updated online in a momentum manner during training. Based on these two modules, the method of this application leverages both the inherent characteristics of the nodule and external knowledge accumulated from other nodules to achieve a reasonable diagnosis. To meet the needs of low-dose and discontinuous screening, large-scale datasets of 12,852 and 4,029 nodules were collected from low-dose and discontinuous CT scans, respectively, each labeled with pathological or subsequent confirmation markers. Experiments on several datasets demonstrate that the method of this application achieves state-of-the-art screening performance in both low-dose and discontinuous scenarios.

[0156] In the above embodiments of this application: by pre-constructing multiple prototypes and the dependency relationship between the first nodule, rich contextual information can be extracted and aggregated from the nodule and its surrounding organs and tissues; by pre-compressing the learned nodule diagnostic knowledge into a prototype and using it as a reference to assist in the diagnosis of new nodules, the final regional features can better match the attributes of the target site itself, and the feature extraction accuracy is higher; based on the features of the first nodule and the features of the second nodule, the diagnosis of benign and malignant lung nodules can be achieved for both low-dose and plain scan screening scenarios, improving the versatility of clinical applications.

[0157] Table 1 presents an ablation comparison of optional hyperparameters according to Embodiment 5 of this application. In Table 1, this application investigates the impact of different configurations on PARE performance on the validation set, including transformer layers, number of prototypes, embedding dimension, and deep supervision. As shown in Table 1, higher AUC scores can be obtained by increasing the number of transformer layers, increasing the number of prototypes, doubling the channel size of token embeddings, or using deep classification supervision. Based on the highest AUC score of 0.931, L=4, N=40, D=256, and DS=true were empirically set in the following experiments. The hyperparameters include: transformer layers (L), number of prototypes (N), embedding dimension (D), and deep supervision (DS).

[0158] Table 1. Comparison of hyperparameters in ablation

[0159] L N D DS AUC 1 20 128 √ 0.912 2 20 128 √ 0.918 4 20 128 √ 0.924 4 10 128 √ 0.920 4 40 128 √ 0.924 4 40 256 √ 0.931 4 40 256 × 0.926

[0160] Table 2 shows the effectiveness of different optional modules according to Embodiment 5 of this application. In Table 2, this application investigated ablation studies of different methods / modules on a validation set and observed the following results: (1) Pure segmentation methods performed better than pure classification methods, mainly because they enabled greater supervision at the pixel level; (2) Joint segmentation and classification outperformed any single method, indicating the complementary effect of the two tasks; (3) Context parsing and prototype comparison both helped improve performance on a strong baseline, thus demonstrating the effectiveness of the two modules; and (4) Segmenting more contextual structures (e.g., blood vessels, lungs, and trachea) provided a slight improvement compared to segmenting nodules alone. Wherein, MT represents multi-task learning. Context: Intra-frame context parsing. Prototype: Inter-prototype review. * indicates that only nodule masks were used in the segmentation task.

[0161] Table 2 Validity of different modules

[0162] Method AUC Pure classification 0.907 Pure segmentation 0.915 MT 0.916 MT+Context* 0.921 MT+Context 0.924 MT+Context+Prototype 0.931

[0163] Comparison with other methods in two screening scenarios: Table 3 compares different methods on an optional NNLST and internal test set according to Embodiment 5 of this application, including pure classification-based methods, pure segmentation-based methods, and multi-task-based methods. Hierarchical evaluation was performed in both test groups based on nodule size distribution. These results indicate that the segmentation-based method outperforms pure classification methods, primarily due to its superior ability to segment the context structure. Furthermore, the multi-task-based CA-Net outperforms any single-task method. On both the NLST and internal test sets, the PARE method of this application outperforms most other methods. Moreover, by utilizing an ensemble of multiple deep supervision heads, the overall AUC is further improved to 0.931 on both datasets. Indicates a pure classification; ◇ represents pure segmentation; ◇ represents multi-task learning; * represents a set of deep supervision heads. Note that this application incorporates a segmentation task into CA-Net.

[0164] Table 3 Comparison of different methods on NNLST and the internal test set

[0165]

[0166] External Evaluation of LUNGx: This application uses LUNGx as an external test to evaluate the generalization of PARE. It is noteworthy that the methods used for these comparisons have never been trained on LUNGx. Table 4 shows an alternative comparison on LUNGx with other methods according to Example 5 of this application. As can be seen from Table 4, the AUC of the PARE model in this application is the highest at 0.801, which is 2% higher than the DAR method. This application also conducted a reader study comparing PARE with two highly experienced radiologists with 8 and 13 years of experience in diagnosing pulmonary nodules, respectively. Figure 10 This is a schematic diagram illustrating an optional comparison between reader research and artificial intelligence according to Embodiment 5 of this application. Figure 3 The results show that the method of this application achieves performance comparable to that of radiologists.

[0167] Table 4 Comparison with other methods on LUNGx

[0168] Method AUC NLNL[9] 0.683 CIRDataset[6] 0.743 D2CNN

[25] 0.746 KBC

[23] 0.768 DAR

[11] 0.781 PARE (Ours) 0.801

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0170] Generalization of LDCT and NCCT: The model in this application was trained on a hybrid of the LDCT and NCCT datasets and performed well in both low-dose and conventional-dose applications. By comparing the generalization performance of the models obtained under three training configurations, the results show that models trained solely on the LDCT or NCCT datasets do not generalize well to other modalities, exhibiting at least a 6% drop in AUC. However, the hybrid training method in this application performs best on both LDCT and NCCT with almost no performance degradation.

[0171] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0172] Example 6

[0173] According to an embodiment of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 11 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 6 of this application, as shown below. Figure 11As shown, the device includes: an acquisition module 1102, a segmentation module 1104, a first determination module 1106, and a second determination module 1108.

[0174] The acquisition module acquires multiple images, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; the segmentation module performs semantic segmentation on the images to obtain the first region features of the monitoring area in the images; the first determination module determines the second region features of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region features, wherein different prototypes are used to characterize different types of monitoring areas; the second determination module identifies the feature information of the monitoring area based on the first region features and the second region features, and determines the identification result of the monitoring area.

[0175] It should be noted that the acquisition module, segmentation module, first determination module, and second determination module mentioned above correspond to steps S302 to S308 in Embodiment 1. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in Embodiment 1.

[0176] In the above embodiments of this application, the segmentation module includes: a segmentation unit, a fusion unit, and a first processing unit.

[0177] The segmentation unit is used to perform semantic segmentation on the image to obtain the semantic segmentation result and global features; the fusion unit is used to fuse the semantic segmentation result and the image features to obtain fused features; and the first processing unit is used to perform attention processing on the global features and fused features to obtain the first region features.

[0178] In the above embodiments of this application, the segmentation unit includes: a first extraction subunit, a second extraction subunit, and a decoding subunit.

[0179] The first extraction subunit is used to extract features from the image using the encoder module of the U-shaped neural network model to obtain the first image features; the second extraction subunit is used to extract global features from the bottleneck layer of the U-shaped neural network model; and the decoding subunit is used to decode the first image features using the encoder module of the U-shaped neural network model to obtain the semantic segmentation result.

[0180] In the above embodiments of this application, the fusion unit includes: a molecular cutting unit, a third extraction subunit, and a fusion subunit.

[0181] The segmentation unit is used to segment the semantic segmentation result and the image respectively, resulting in multiple sub-segmentation results and multiple sub-images; the third extraction unit is used to extract features from the multiple sub-segmentation results and multiple sub-images respectively, resulting in sub-segmentation features of the multiple sub-segmentation results and sub-image features of the multiple sub-images; the fusion unit is used to fuse the sub-segmentation features and sub-image features to obtain fused features.

[0182] In the above embodiments of this application, the first processing unit includes: a splicing subunit and a processing subunit.

[0183] The splicing subunit is used to splice the global features and the fused features to obtain the first spliced ​​feature; the processing subunit is used to perform self-attention processing on the first spliced ​​feature using a self-attention model to obtain the first region feature.

[0184] In the above embodiments of this application, the first determining module includes: a second processing unit.

[0185] The second processing unit is used to perform attention processing on the first region features and multiple prototypes using a cross-attention model to obtain the second region features.

[0186] In the above embodiments of this application, the first determining module further includes: an acquisition unit, a clustering unit, and a construction unit.

[0187] The acquisition unit is used to acquire global features of different monitoring areas; the clustering unit is used to cluster the global features of different monitoring areas to obtain multiple feature sets; and the construction unit is used to construct multiple prototypes based on the central features of multiple feature sets.

[0188] In the above embodiments of this application, after determining the second regional features of the monitoring area based on the dependency relationship between multiple pre-built prototypes and the first regional features, the device further includes: a third determining module, a first updating module, and a second updating module.

[0189] The third determining module is used to determine the target prototype that successfully matches the second region features from multiple pre-built prototypes; the first updating module is used to perform momentum updates on the target prototype to obtain updated region features; and the second updating module is used to update multiple prototypes based on the updated region features.

[0190] In the above embodiments of this application, the second determining module includes: a third processing unit, a fourth processing unit, a fifth processing unit, and a summarizing unit.

[0191] The third processing unit is used to identify the feature information of the monitoring area based on global features to obtain the first sub-identification result; the fourth processing unit is used to identify the feature information of the monitoring area based on the first region features to obtain the second sub-identification result; the fifth processing unit is used to identify the feature information of the monitoring area based on the second region features to obtain the third sub-identification result; and the summarization unit is used to summarize the first sub-identification result, the second sub-identification result and the third sub-identification result to obtain the identification result.

[0192] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0193] Example 7

[0194] According to an embodiment of this application, an image processing apparatus for implementing the above-described image processing method is also provided.

[0195] Figure 12 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 7 of this application, as shown below. Figure 12 As shown, the device includes: a first display module 1202 and a second display module 1204.

[0196] The first display module is used to respond to input commands applied to the operation interface and display multiple images on the operation interface. The displayed content of the images includes at least the monitoring area of ​​the target part of the object to be monitored. The second display module is used to respond to image processing commands applied to the operation interface and display the recognition result of the monitoring area on the operation interface. The recognition result is obtained by recognizing the feature information of the monitoring area based on the first region feature and the second region feature. The second region feature is determined based on the dependency relationship between multiple pre-constructed prototypes and the first region feature. The first region feature is obtained by semantic segmentation of the medical image.

[0197] It should be noted that the first display module and the second display module mentioned above correspond to steps S502 to S504 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in Embodiment 1.

[0198] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0199] Example 8

[0200] According to an embodiment of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 13 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 8 of this application, as shown below. Figure 13 As shown, the device includes: a display module 1302, a segmentation module 1304, a first determination module 1306, a second determination module 1308, and a driving module 13010.

[0201] The system comprises the following modules: a display module for displaying multiple images on the screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the displayed content of the images includes at least the monitoring area of ​​the target part of the object to be monitored; a segmentation module for performing semantic segmentation on the images to obtain the first region features of the monitoring area in the images; a first determination module for determining the second region features of the monitoring area based on the dependency relationship between multiple pre-built prototypes and the first region features, wherein different prototypes are used to characterize different types of monitoring areas; a second determination module for identifying the feature information of the monitoring area based on the first region features and the second region features, and determining the identification result of the monitoring area; and a driving module for driving the VR or AR device to render and display the identification result.

[0202] It should be noted that the display module, segmentation module, first determining module, second determining module, and driving module mentioned above correspond to steps S702 to S7010 in Embodiment 3. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in Embodiment 1.

[0203] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0204] Example 9

[0205] According to an embodiment of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 14 This is a schematic diagram of the structure of an image processing apparatus according to Embodiment 9 of this application, as shown below. Figure 14 As shown, the device includes: an acquisition module 1402, a segmentation module 1404, a first determination module 1406, a second determination module 1408, and an output module 14010.

[0206] The system comprises the following modules: an acquisition module for acquiring multiple images by calling a first interface, wherein the first interface includes a first parameter whose value is multiple images, and the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; a segmentation module for performing semantic segmentation on the images to obtain the first region features of the monitoring area in the images; a first determination module for determining the second region features of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region features, wherein different prototypes are used to represent different types of monitoring areas; a second determination module for identifying the feature information of the monitoring area based on the first region features and the second region features, and determining the identification result of the monitoring area; and an output module for outputting the identification result by calling a second interface, wherein the second interface includes a second parameter whose value is the identification result.

[0207] It should be noted that the acquisition module, segmentation module, first determination module, second determination module, and output module mentioned above correspond to steps S802 to S8010 in Embodiment 4. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 4. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in Embodiment 1.

[0208] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0209] Example 10

[0210] According to an embodiment of this application, a lung nodule diagnostic device for implementing the above-described lung nodule diagnostic method is also provided. Figure 15 This is a schematic diagram of the structure of a lung nodule diagnostic device according to Embodiment 10 of this application, as shown below. Figure 15 As shown, the device includes: an acquisition module 1502, a segmentation module 1504, a determination module 1506, and a diagnosis module 1508.

[0211] The system comprises the following modules: an acquisition module for acquiring multiple medical images containing lung nodules; a segmentation module for semantic segmentation of the medical images to obtain the first nodule features of the lung nodules; a determination module for determining the second nodule features of the lung nodules based on the dependencies between multiple pre-built prototypes and the first nodule features, where different prototypes characterize different types of lung nodules; and a diagnosis module for diagnosing the lung nodules based on the first and second nodule features to obtain a diagnostic result, where the diagnostic result characterizes whether the lung nodule is benign or malignant.

[0212] It should be noted that the acquisition module, segmentation module, determination module, and diagnosis module mentioned above correspond to steps S902 to S908 in Embodiment 5. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 5. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in Embodiment 1.

[0213] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0214] Example 11

[0215] Embodiments of this application may provide an AR / VR device, which can be any AR / VR device from a group of AR / VR devices. Optionally, in this embodiment, the aforementioned AR / VR device may also be replaced by a terminal device such as a mobile terminal.

[0216] Optionally, in this embodiment, the AR / VR device described above may be located in at least one of a plurality of network devices in a computer network.

[0217] In this embodiment, the AR / VR device described above can execute the program code for the following steps in the image processing method: displaying multiple images on the presentation screen of the virtual reality (VR) device or augmented reality (AR) device, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; performing semantic segmentation on the images to obtain the first region features of the monitoring area in the images; determining the second region features of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region features, wherein different prototypes are used to characterize different types of monitoring areas; identifying the feature information of the monitoring area based on the first region features and the second region features, and determining the identification result of the monitoring area; driving the VR device or AR device to render and display the identification result.

[0218] Optionally, Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 16 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1602, memory 1604, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0219] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring multiple images, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; performing semantic segmentation on the images to obtain first region features of the monitoring area in the images; determining second region features of the monitoring area based on the dependencies between multiple pre-constructed prototypes and the first region features, wherein different prototypes are used to characterize different types of monitoring areas; identifying feature information of the monitoring area based on the first region features and the second region features, and determining the identification result of the monitoring area.

[0221] Optionally, the processor may also execute program code that performs the following steps: semantic segmentation of the image to obtain the semantic segmentation result and global features of the image; feature fusion of the semantic segmentation result and the image to obtain fused features; attention processing of the global features and fused features to obtain first region features.

[0222] Optionally, the processor may also execute program code that performs the following steps: extracting features from the image using the encoder module of the U-shaped neural network model to obtain the first image features; extracting global features from the bottleneck layer of the U-shaped neural network model; and decoding the first image features using the encoder module of the U-shaped neural network model to obtain the semantic segmentation result.

[0223] Optionally, the processor may also execute program code that performs the following steps: segmenting the semantic segmentation result and the image separately to obtain multiple sub-segmentation results and multiple sub-images; extracting features from the multiple sub-segmentation results and multiple sub-images to obtain sub-segmentation features of the multiple sub-segmentation results and sub-image features of the multiple sub-images; and fusing the sub-segmentation features and sub-image features to obtain fused features.

[0224] Optionally, the processor may also execute program code that performs the following steps: concatenating global features and fused features to obtain a first concatenated feature; and performing self-attention processing on the first concatenated feature using a self-attention model to obtain a first region feature.

[0225] Optionally, the processor may also execute program code that performs attention processing on the first region features and multiple prototypes using a cross-attention model to obtain the second region features.

[0226] Optionally, the processor may also execute program code that performs the following steps: acquiring global features of different monitoring areas; clustering the global features of different monitoring areas to obtain multiple feature sets; and constructing multiple prototypes based on the central features of the multiple feature sets.

[0227] Optionally, the processor may also execute program code that performs the following steps: determining a target prototype that successfully matches the second region features from a plurality of pre-built prototypes; performing momentum updates on the target prototype to obtain updated region features; and updating the plurality of prototypes based on the updated region features.

[0228] Optionally, the processor may also execute program code that performs the following steps: based on global features, identify the feature information of the monitoring area to obtain a first sub-identification result; based on the first region features, identify the feature information of the monitoring area to obtain a second sub-identification result; based on the second region features, identify the feature information of the monitoring area to obtain a third sub-identification result; and summarize the first sub-identification result, the second sub-identification result, and the third sub-identification result to obtain the identification result.

[0229] This application provides a method for acquiring multiple images; performing semantic segmentation on the images to obtain first region features of a monitored area in the images; determining second region features of the monitored area based on the dependency relationship between multiple pre-constructed prototypes and the first region features; and identifying feature information of the monitored area based on the first and second region features to determine the identification result of the monitored area. It is noteworthy that this application not only performs semantic segmentation on the images to obtain region features, but also combines the dependency relationship between prototypes and region features to ensure that the final region features better match the attributes of the target part itself, resulting in higher feature extraction accuracy. This achieves the goal of more accurately identifying the target part of the monitored object, thereby improving the technical effect of increasing the identification accuracy of the target part of the monitored object and solving the technical problem of low identification accuracy in related technologies for identifying monitored images.

[0230] Those skilled in the art will understand that the structure shown in the figure is for illustrative purposes only, and the computer terminal may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 16 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more complex than those described above. Figure 16 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 16 The different configurations shown.

[0231] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0232] Example 12

[0233] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image processing method provided in Embodiment 1.

[0234] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in the AR / VR device terminal group in the AR / VR device network, or in any mobile terminal in the mobile terminal group.

[0235] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring multiple images, wherein the displayed content of the images at least includes the monitoring area of ​​the target part of the object to be monitored; performing semantic segmentation on the images to obtain a first region feature of the monitoring area in the images; determining a second region feature of the monitoring area based on the dependency relationship between multiple pre-constructed prototypes and the first region feature, wherein different prototypes are used to characterize different types of monitoring areas; identifying the feature information of the monitoring area based on the first region feature and the second region feature, and determining the identification result of the monitoring area.

[0236] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: performing semantic segmentation on the image to obtain semantic segmentation results and global features of the image; performing feature fusion on the semantic segmentation results and the image to obtain fused features; and performing attention processing on the global features and fused features to obtain first region features.

[0237] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: extracting features from the image using the encoder module of the U-shaped neural network model to obtain the first image features of the image; extracting global features from the bottleneck layer of the U-shaped neural network model; and decoding the first image features using the encoder module of the U-shaped neural network model to obtain a semantic segmentation result.

[0238] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: segmenting the semantic segmentation result and the image respectively to obtain multiple sub-segmentation results and multiple sub-images; extracting features from the multiple sub-segmentation results and multiple sub-images respectively to obtain sub-segmentation features of the multiple sub-segmentation results and sub-image features of the multiple sub-images; and fusing the sub-segmentation features and sub-image features to obtain fused features.

[0239] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: concatenating global features and fused features to obtain a first concatenated feature; and performing self-attention processing on the first concatenated feature using a self-attention model to obtain a first region feature.

[0240] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: performing attention processing on the first region features and multiple prototypes using a cross-attention model to obtain the second region features.

[0241] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: obtaining global features of different monitoring areas; clustering the global features of different monitoring areas to obtain multiple feature sets; and constructing multiple prototypes based on the central features of the multiple feature sets.

[0242] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: determining a target prototype that successfully matches the second region features from a plurality of pre-built prototypes; performing momentum updates on the target prototype to obtain updated region features; and updating the plurality of prototypes based on the updated region features.

[0243] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: identifying feature information of the monitoring area based on global features to obtain a first sub-identification result; identifying feature information of the monitoring area based on the features of the first region to obtain a second sub-identification result; identifying feature information of the monitoring area based on the features of the second region to obtain a third sub-identification result; and summarizing the first sub-identification result, the second sub-identification result, and the third sub-identification result to obtain an identification result.

[0244] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0245] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0246] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0247] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0249] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0250] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: acquiring a plurality of images, wherein the display content of the images at least contains a monitoring area of a target part of an object to be monitored; performing semantic segmentation on the images to obtain first regional features of the monitoring area in the images; determining second regional features of the monitoring area based on a dependency relationship between a plurality of prototypes constructed in advance and the first regional features, wherein different prototypes are used to represent different types of monitoring areas; identifying feature information of the monitoring area based on the first regional features and the second regional features to determine an identification result of the monitoring area; wherein determining the second regional features of the monitoring area based on the dependency relationship between the plurality of prototypes constructed in advance and the first regional features comprises performing attention processing on the first regional features and the plurality of prototypes to obtain the second regional features.

2. The method of claim 1, wherein, The method comprises the following steps: performing semantic segmentation on the images to obtain first regional features of the monitoring area in the images, comprising: performing semantic segmentation on the images to obtain semantic segmentation results and global features of the images; performing feature fusion on the semantic segmentation results and the images to obtain fusion features; 3. The method of claim 2, wherein, performing attention processing on the global features and the fusion features to obtain the first regional features. The method comprises the following steps: performing feature extraction on the images by using an encoder module of a U-shaped neural network model to obtain first image features of the images; extracting the global features from a bottleneck layer of the U-shaped neural network model; 4. The method of claim 2, wherein, decoding the first image features by using a decoder module of the U-shaped neural network model to obtain the semantic segmentation results. The method comprises the following steps: respectively segmenting the semantic segmentation results and the images to obtain a plurality of sub-segmentation results and a plurality of sub-images; respectively performing feature extraction on the plurality of sub-segmentation results and the plurality of sub-images to obtain sub-segmentation features of the plurality of sub-segmentation results and sub-image features of the plurality of sub-images; 5. The method of claim 2, wherein, fusing the sub-segmentation features and the sub-image features to obtain the fusion features. The method comprises the following steps: splicing the global features and the fusion features to obtain first splicing features; 6. The method of claim 1, wherein, performing self-attention processing on the first splicing features by using a self-attention model to obtain the first regional features. The method comprises the following steps:

7. The method of claim 6, wherein, performing attention processing on the first regional features and the plurality of prototypes by using a cross-attention model to obtain the second regional features. The method further comprises the following steps: acquiring global features of different monitoring areas; clustering the global features of the different monitoring areas to obtain a plurality of feature sets; constructing the plurality of prototypes based on center features of the plurality of feature sets.

8. The method of claim 1, wherein, After determining the second region feature of the monitoring region based on the dependency relationship between the plurality of pre-constructed prototypes and the first region feature, the method further comprises: determining a target prototype matched with the second region feature from the plurality of pre-constructed prototypes; updating the target prototype to obtain an updated region feature; updating the plurality of prototypes based on the updated region feature.

9. The method of claim 1, wherein, identifying the feature information of the monitoring region based on the first region feature and the second region feature, and determining an identification result of the monitoring region, comprising: identifying the feature information of the monitoring region based on a global feature to obtain a first sub-identification result, wherein the global feature is obtained by performing semantic segmentation on the image; identifying the feature information of the monitoring region based on the first region feature to obtain a second sub-identification result; identifying the feature information of the monitoring region based on the second region feature to obtain a third sub-identification result; summarizing the first sub-identification result, the second sub-identification result and the third sub-identification result to obtain the identification result.

10. An image processing method characterized by, comprising: in response to an input instruction acting on an operation interface, displaying a plurality of images on the operation interface, wherein the display content of the image at least contains a monitoring region of a target part of a to-be-monitored object; in response to an image processing instruction acting on the operation interface, displaying an identification result of the monitoring region on the operation interface, wherein the identification result is obtained based on identifying the feature information of the monitoring region based on a first region feature and a second region feature, the second region feature is obtained by performing attention processing on a plurality of pre-constructed prototypes and the first region feature, and the first region feature is obtained by performing semantic segmentation on a medical image.

11. An image processing method, characterized by, comprising: displaying a plurality of images on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the display content of the image at least contains a monitoring region of a target part of a to-be-monitored object; performing semantic segmentation on the image to obtain a first region feature of the monitoring region in the image; determining a second region feature of the monitoring region based on a dependency relationship between a plurality of pre-constructed prototypes and the first region feature, wherein different prototypes are used to represent different types of monitoring regions; identifying the feature information of the monitoring region based on the first region feature and the second region feature, and determining an identification result of the monitoring region; driving the VR device or the AR device to render and display the identification result; wherein determining the second region feature of the monitoring region based on the dependency relationship between the plurality of pre-constructed prototypes and the first region feature comprises performing attention processing on the first region feature and the plurality of prototypes to obtain the second region feature.

12. An image processing method, characterized by, comprising: obtaining a plurality of images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is the plurality of images, and the display content of the image at least contains a monitoring region of a target part of a to-be-monitored object; performing semantic segmentation on the image to obtain a first region feature of the monitoring region in the image; determining a second region feature of the monitoring region based on a dependency relationship between a plurality of prototypes constructed in advance and the first region feature, wherein different prototypes are used to represent different types of monitoring regions; identifying feature information of the monitoring region based on the first region feature and the second region feature, and determining an identification result of the monitoring region; outputting the identification result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the identification result; wherein determining the second region feature of the monitoring region based on the dependency relationship between the plurality of prototypes constructed in advance and the first region feature includes performing attention processing on the first region feature and the plurality of prototypes to obtain the second region feature.

13. An electronic device, comprising: comprise: a memory storing an executable program; a processor configured to execute the program, wherein the program performs the method of any one of claims 1 to 12 when executed.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored executable program, wherein the computer-readable storage medium controls the device where the computer-readable storage medium is located to perform the method of any one of claims 1 to 12 when the executable program is executed.

Images