Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and the method is applied to the field of image processing, and comprises the steps: obtaining N feature maps of a to-be-processed image outputted by a semantic segmentation model; n is greater than 1, each feature map corresponds to one category, and the resolutions of the N feature maps are all smaller than a preset resolution threshold; copying the N feature maps into a memory; pointers corresponding to the N feature maps are controlled to synchronously move in the N feature maps in the memory, and feature data pointed by the N pointers are obtained; according to the feature data, determining a semantic segmentation result of a first resolution corresponding to the to-be-processed image; wherein the semantic segmentation result comprises a category corresponding to each pixel position, and the first resolution is smaller than a preset resolution threshold. According to the method, the resource occupation amount and the calculation amount can be reduced in the post-processing process of semantic segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Semantic segmentation is a fundamental but crucial task in computer vision. Its goal is to predict a specific category for each pixel location in an image, enabling a more comprehensive and thorough understanding of the information contained in the entire image scene.

[0003] Currently, the post-processing methods involved in applying semantic segmentation technology to the device side include: upsampling each feature map output by the semantic segmentation model, then calculating the maximum probability of each pixel location in each upsampled feature map, and using the category corresponding to the maximum probability as the category of each pixel location. However, this method suffers from high resource consumption and computational complexity. Therefore, how to reduce resource consumption and computational complexity in the post-processing of semantic segmentation has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and storage medium, which can reduce resource consumption and computational load during the post-processing of semantic segmentation.

[0005] In a first aspect, an image processing method is provided, comprising: acquiring N feature maps of an image to be processed output by a semantic segmentation model; wherein N > 1, each feature map corresponds to a category, and the resolution of each of the N feature maps is less than a preset resolution threshold; copying the N feature maps into memory; controlling the pointers corresponding to each of the N feature maps to move synchronously in the N feature maps in memory, thereby obtaining feature data pointed to by each of the N pointers; wherein the feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to the feature map; and determining a semantic segmentation result of a first resolution corresponding to the image to be processed based on the feature data; wherein the semantic segmentation result includes the category corresponding to each pixel position, and the first resolution is less than the preset resolution threshold.

[0006] The above technical solution directly copies the N feature maps of the image to be processed output by the semantic segmentation model into memory, without needing to upsample all N feature maps before copying them into memory. Since the resolution of these N feature maps is all less than a preset resolution threshold (i.e., the N feature maps have low resolution), directly copying the N feature maps output by the semantic segmentation model into memory helps save data copying, thereby reducing memory usage and the time required for data copying. Furthermore, determining the semantic segmentation result of the image to be processed at the first resolution based on the low-resolution feature maps requires less computation than determining the category corresponding to each pixel position based on high-resolution feature maps, thus reducing computational load. Simultaneously, setting pointers for each of the N feature maps allows for continuous access to the feature data of each feature map through its corresponding pointer. This pointer does not need to jump between different feature maps, thus reducing latency caused by skipping access and avoiding invalid addressing caused by skipping access. This reduces power consumption, shortens computation time, and improves computational efficiency in the post-processing of semantic segmentation.

[0007] In conjunction with the first aspect, in some possible implementations, each of the aforementioned pointers is used to point to a storage address of one of the aforementioned feature maps in the aforementioned memory, and the aforementioned N feature maps have the same resolution; controlling the pointers corresponding to each of the aforementioned N feature maps to move synchronously in the N feature maps in the aforementioned memory to obtain the feature data pointed to by each of the N pointers includes: controlling the pointers corresponding to each of the aforementioned N feature maps to move synchronously in the storage addresses corresponding to the N feature maps in the aforementioned memory; each time the N pointers move to the storage address corresponding to the target pixel position in the aforementioned N feature maps, reading the feature data in the storage address pointed to by the N pointers respectively; wherein, the pixel coordinates corresponding to the target pixel positions in the N feature maps are the same, and the aforementioned feature data includes the probability that the target pixel position in each of the aforementioned feature maps belongs to the category corresponding to the aforementioned feature map.

[0008] In the above technical solution, by setting an independent pointer for each feature map, the data of the target pixel position in N feature maps can be read simultaneously as the N pointers move synchronously. This helps to accelerate the execution speed of the argmax operation, that is, to quickly determine the category corresponding to the target pixel position, thereby further improving the computational efficiency in the post-processing of semantic segmentation.

[0009] In conjunction with the first aspect, in some possible implementations, controlling the pointers corresponding to the N feature maps to move synchronously within the storage addresses corresponding to the N feature maps in the memory includes: controlling the N pointers to move to their respective starting addresses; wherein the starting addresses of the N pointers correspond to the initial pixel positions of the N feature maps, and the pixel coordinates corresponding to the initial pixel positions of the N feature maps are the same; and controlling the N pointers to move synchronously from their respective starting addresses according to a preset pointer movement direction and pointer movement offset.

[0010] In the above technical solution, by setting the starting addresses of N pointers to correspond to their respective initial pixel positions, the starting coordinates of reading all feature map data are consistent. This helps to accurately synchronize the reading process of different feature maps and ensures the accuracy of subsequent data processing. The preset pointer movement direction and offset ensure that the N pointers move according to the same rules, guaranteeing the consistency of pixel coordinates in the data read from different feature maps. By setting explicit starting addresses and unified movement rules for the N pointers, the logic of pointer control is simplified, and the possibility of errors is reduced.

[0011] In conjunction with the first aspect, in some possible implementations, after determining the semantic segmentation result of the first resolution corresponding to the image to be processed based on the feature data, the method further includes: obtaining a mask image of a specified category based on the semantic segmentation result of the first resolution; determining the feature map corresponding to the specified category among the N feature maps; determining the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category; and determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the target feature sub-map; wherein the second resolution is greater than the first resolution.

[0012] In the above technical solution, based on actual needs, the semantic segmentation result at the first resolution can acquire only the mask image of the specified category, and further processing can be performed on the specified category without further processing on all categories, which helps to reduce the amount of computation. Further processing based on the mask image and feature map of the specified category yields a target feature sub-map, which integrates the feature information of the mask image and the feature map of the specified category. This ensures that the semantic segmentation result contains both global information and local details, enhancing the comprehensiveness and accuracy of the segmentation result, and resulting in higher precision for the semantic segmentation result obtained for the specified category.

[0013] In conjunction with the first aspect, in some possible implementations, determining the target feature sub-image corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category includes: upsampling the mask image and the feature map corresponding to the specified category respectively to obtain a first upsampled image of the mask image and a second upsampled image of the feature map corresponding to the specified category; determining the bounding box corresponding to the pixel position of the specified category on the first upsampled image; and extracting the feature sub-image corresponding to the specified category from the second upsampled image based on the bounding box, and using the feature sub-image corresponding to the specified category as the target feature sub-image.

[0014] In the above technical solution, upsampling is performed on the mask image and the feature map corresponding to the specified category. Since the upsampled image has higher image detail and richness, combining the first and second upsampled images helps improve the accuracy of the semantic segmentation result for the specified category, reducing jagged edges at the segmentation boundaries and making the segmentation result appear smoother and more natural. Simultaneously, since the upsampling only applies to the mask image and the feature map corresponding to the specified category, upsampling is not required for feature maps other than the specified category. Therefore, this upsampling operation will not significantly impact the computational efficiency in the post-processing of semantic segmentation. Furthermore, since the first upsampled image is an upsampled version of the mask image for the specified category, it is easy to accurately obtain the bounding box corresponding to the pixel position of the specified category, thereby initially determining the area where the specified category is distributed in the first upsampled image, i.e., the area covered by the bounding box. Then, based on the bounding box, a feature sub-image corresponding to the specified category is extracted from the second upsampled image. Semantic segmentation is performed based on this feature sub-image. Compared with performing semantic segmentation on the entire feature map, this can significantly reduce the amount of computation and help to further improve computational efficiency.

[0015] In combination with the first aspect and the above implementation, in some possible implementations, determining the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category includes: determining the bounding box corresponding to the pixel position of the specified category on the mask image; extracting the feature sub-map corresponding to the specified category from the feature map corresponding to the specified category based on the bounding box; upsampling the feature sub-map to obtain a third upsampled image, and using the third upsampled image as the target feature sub-map.

[0016] In the above technical solution, only the extracted feature sub-image needs to be upsampled, instead of the full-scale feature image, which greatly reduces the computational load of upsampling. Furthermore, the third upsampled image can reflect richer details related to the specified category, thus improving the accuracy of the semantic segmentation result at the second resolution corresponding to the specified category. Upsampling also reduces jagged edges on the segmentation boundaries, making the segmentation result appear smoother and more natural.

[0017] In combination with the first aspect and the above implementation, in some possible implementations, determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the target feature sub-image includes: dividing the target feature sub-image into foreground and background to obtain the foreground region in the target feature sub-image; and determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the foreground region in the target feature sub-image.

[0018] In combination with the first aspect and the above implementation, in some possible implementations, determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the foreground region in the target feature sub-image includes: mapping the foreground region in the target feature sub-image to the image to be processed to obtain the mapped region of the foreground region in the image to be processed; and determining the mapped region as the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0019] Secondly, an image processing apparatus is provided, comprising: an acquisition module for acquiring N feature maps of an image to be processed output by a semantic segmentation model; wherein N > 1, each feature map corresponds to a category, and the resolution of each of the N feature maps is less than a preset resolution threshold; a copying module for copying the N feature maps into memory; a control module for controlling pointers corresponding to each of the N feature maps to move synchronously among the N feature maps in memory, thereby obtaining feature data pointed to by each of the N pointers; wherein the feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to the feature map; and a determination module for determining a semantic segmentation result at a first resolution corresponding to the image to be processed based on the feature data; wherein the semantic segmentation result includes the category corresponding to each pixel position, and the first resolution is less than the preset resolution threshold.

[0020] Thirdly, an electronic device is provided, comprising: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, causing the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0021] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0022] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the principle of post-processing methods involved when semantic segmentation technology is applied to the device side in related technologies;

[0024] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a processing flow that is included after determining the semantic segmentation result of the first resolution corresponding to the image to be processed according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a mask image corresponding to a specified category provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an external bounding box provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram illustrating the principle of an image processing method provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0033] Semantic segmentation is a fundamental but crucial task in computer vision. Its goal is to predict a specific category for each pixel location in an image, achieving pixel-level classification of the image, which facilitates a more comprehensive and thorough understanding of the information contained in the entire image scene.

[0034] Currently, a schematic diagram illustrating the post-processing methods involved in applying semantic segmentation technology to the device side can be found in [reference needed]. Figure 1 ,include:

[0035] First, the image to be processed is input into a semantic segmentation model, which outputs multiple low-resolution feature maps. Next, these low-resolution feature maps are upsampled to obtain multiple high-resolution feature maps. Then, these high-resolution feature maps are copied into memory. Based on these memory-based high-resolution feature maps, the maximum probability of each pixel location is calculated, and the category corresponding to the highest probability is used as the category for each pixel location to obtain the semantic segmentation result.

[0036] This application, through research, has found that Figure 1 In the post-processing method shown, copying multiple upsampled high-resolution feature maps into memory significantly increases memory usage. Furthermore, calculating the maximum probability for each pixel location based on the high-resolution feature maps introduces a large computational load, resulting in high overall latency. The high overall latency is also due to the need to access data on different feature maps in a skipping manner when using a single pointer; memory skipping leads to additional latency and power consumption, resulting in low computational efficiency. Therefore, Figure 1 The post-processing method shown suffers from high memory consumption and computational complexity. Therefore, reducing resource consumption and computational complexity during semantic segmentation post-processing has become an urgent technical problem to be solved.

[0037] Based on this, to at least solve the aforementioned technical problems, embodiments of this application provide an image processing method aimed at reducing memory resource consumption by avoiding copying all upsampled high-resolution feature maps in memory. Furthermore, it reduces computational complexity by performing an argmax operation based on low-resolution feature maps. The argmax operation involves calculating the maximum probability for each pixel location based on N feature maps, and using the category corresponding to the maximum probability as the category for each pixel location to obtain semantic segmentation results. Simultaneously, by setting pointers for each of the N feature maps, additional latency and power consumption caused by memory jumps are avoided.

[0038] The image processing method in this embodiment is applied to an electronic device, which is an edge device, and a semantic segmentation model is deployed in the edge device. This semantic segmentation model can be any of the following: Bisnet (Bilateral Segmentation Network), FCN (Fully Convolutional Networks), ENet (Efficient Neural Network), or OCRNet (Object Contextual Representations). However, this embodiment does not specifically limit the specific form of the semantic segmentation model.

[0039] The aforementioned edge devices possess certain computing, storage, and communication capabilities, enabling them to process data and perform specific tasks, such as semantic segmentation. For example, edge devices can be: smartphones, tablets, smartwatches, smart bracelets, and other mobile devices used daily by users; IoT devices; interactive conference tablets; drones; autonomous vehicles, and other devices that may involve semantic segmentation tasks.

[0040] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0041] For example, such as Figure 2 As shown, the image processing method includes:

[0042] Step 201: Obtain N feature maps of the image to be processed from the output of the semantic segmentation model.

[0043] Step 202: Copy the N feature maps into memory.

[0044] Step 203: Control the pointers corresponding to the N feature maps to move synchronously in the N feature maps in memory, and obtain the feature data pointed to by the N pointers respectively.

[0045] Step 204: Determine the semantic segmentation result of the first resolution corresponding to the image to be processed based on the feature data.

[0046] exist Figure 2 In the illustrated embodiment, the N feature maps of the image to be processed output by the semantic segmentation model are directly copied into memory, without the need to upsample all N feature maps before copying them into memory. Since the resolution of these N feature maps is all less than a preset resolution threshold (i.e., the N feature maps have low resolution), directly copying the N feature maps output by the semantic segmentation model into memory saves data copying, thereby reducing memory usage and the time required for data copying. Furthermore, determining the semantic segmentation result of the image to be processed at a first resolution based on the low-resolution feature maps requires less computation than determining the category corresponding to each pixel position based on high-resolution feature maps, thus reducing computational load. Simultaneously, setting pointers for each of the N feature maps allows for continuous access to the feature data of each feature map through its corresponding pointer. This pointer does not need to jump between different feature maps, thus reducing latency caused by skipping access and avoiding invalid addressing caused by skipping access. This reduces power consumption, shortens computation time, and improves computational efficiency in the post-processing of semantic segmentation.

[0047] The following is about Figure 2 The specific implementation methods of each step in the illustrated embodiment are explained below:

[0048] In step 201, the image to be processed is the image to be segmented. The end device is equipped with an image processing device. The image processing method in this embodiment can be specifically executed by the image processing device configured in the electronic device.

[0049] Specifically, the image processing device first acquires the image to be processed, then inputs it into a semantic segmentation model. The semantic segmentation model processes the input image and outputs N feature maps, where N > 1. Each feature map corresponds to a category, and the N feature maps correspond to N categories. All N feature maps have the same size and resolution. These N categories include the categories of all objects present in the image. Each feature map can be understood as a probability map corresponding to a category, representing the probability that each pixel in the feature map belongs to that category. The resolution of all N feature maps is less than a preset resolution threshold, which is lower than the resolution of the image to be processed. Therefore, compared to the original image, the N feature maps are low-resolution feature maps.

[0050] For example, if the objects to be processed include three categories of objects: clouds, houses, and trees, then the semantic segmentation model will output three feature maps, including the features corresponding to clouds. Figure 1 Features of the house Figure 2 Features corresponding to trees Figure 3 Characteristics corresponding to clouds Figure 1 Characteristic features Figure 1 The probability that each pixel location belongs to the cloud category. Features corresponding to houses. Figure 2 Characteristic features Figure 2 The probability that each pixel location belongs to the "house" category, and the corresponding features of the tree. Figure 3 Characteristic features Figure 3 The probability that each pixel location belongs to the tree category.

[0051] It should be noted that the feature maps output by the semantic segmentation model are usually low-resolution feature maps, and their size is generally 1 / 8 or 1 / 16 of the size of the image to be processed, i.e., the original input image. For ease of description, in this embodiment, the resolution of the feature map output by the semantic segmentation model is denoted as the first resolution, which is less than the aforementioned preset resolution threshold.

[0052] In step 202, the image processing device can copy the N feature maps to the memory of the edge device, specifically by copying the N feature maps from the video memory of the edge device to the main memory. As mentioned earlier, these N feature maps are directly output by the semantic segmentation model and have not been upsampled; they are low-resolution feature maps and occupy little storage space. Therefore, copying these N feature maps to the main memory can reduce the consumption of memory resources.

[0053] In steps 203 and 204, the image processing device performs an argmax operation based on the N feature maps copied to memory to determine the category corresponding to each pixel position in the feature maps, thereby obtaining a semantic segmentation result at a first resolution for the image to be processed. The first resolution is the resolution of the feature maps output by the semantic segmentation model. Since the resolution of the feature maps directly output by the semantic segmentation model is usually low, the semantic segmentation result at the first resolution can also be understood as a low-resolution semantic segmentation result.

[0054] As mentioned earlier, each feature map can be understood as a probability map corresponding to a category. In semantic segmentation tasks, the argmax operation is used to classify each pixel position based on the probability map output by the semantic segmentation model, and to determine the category to which each pixel position belongs, that is, the category corresponding to each pixel position.

[0055] For example, based on the example above, the semantic segmentation model outputs three feature maps, including: features corresponding to the clouds. Figure 1 Features of the house Figure 2 Features corresponding to trees Figure 3Therefore, combining these three feature maps, each pixel location actually corresponds to three probability values: the probability that the pixel location belongs to a cloud, the probability that the pixel location belongs to a house, and the probability that the pixel location belongs to a tree. If, through comparison, it is determined that the probability of the pixel location belonging to a house is the highest, then the category corresponding to the pixel location is determined to be a house. In other words, for each pixel location, the argmax operation selects the highest probability among the above three probabilities and classifies the pixel location into the category corresponding to the highest probability.

[0056] For example, after performing the argmax operation, the category corresponding to each pixel position in the feature map is obtained. Based on this, a category distribution map can be generated, in which each pixel position is labeled with its corresponding category. For example, in this category distribution map, the pixel position belonging to the cloud is labeled 1, the pixel position belonging to the house is labeled 2, and the pixel position belonging to the tree is labeled 3. It should be noted that the labels 1, 2, and 3 here are only for illustrative purposes. In specific implementations, other labels can be used, and this embodiment does not impose any specific limitations on this.

[0057] exist Figure 1 In the illustrated embodiment, a pointer Ptr0 is set when calculating the maximum probability for each pixel location. Since the N feature maps are stored contiguously in memory, setting a single pointer Ptr0 presents the problem of needing to access feature data at different storage addresses in a skip-like manner. Memory skipping leads to additional latency and power consumption. For ease of understanding, a specific example is provided below:

[0058] Referring to the example above, the N feature maps include the features corresponding to category 1. Figure 1 Features corresponding to category 2 Figure 2 Features corresponding to category 3 Figure 3 Assumptions, features Figure 1 The memory addresses are: address 1 to address 100, characteristics Figure 2 The memory addresses are from 101 to 200, and the characteristics are... Figure 3 The memory addresses are from 201 to 300. Therefore, when executing the argmax operation, the pointer Ptr0 first points to address 1 to read the characteristics. Figure 1 The feature data for pixel position 1 (including the probability that pixel position 1 belongs to category 1) is read, and then pointer Ptr0 jumps from address 1 to address 101 (requiring 100 jumps) to read the feature. Figure 2 The feature data for pixel position 1 (including the probability that pixel position 1 belongs to category 2) is read, and then pointer Ptr0 needs to jump from address 101 to address 201 (requiring 100 jumps) to read the feature data. Figure 3The feature data for pixel position 1 (this feature data includes the probability that pixel position 1 belongs to category 3). Finally, the read features can be... Figure 1 Feature data and features of pixel position 1 Figure 2 Feature data and features of pixel position 1 Figure 3 The feature data at pixel position 1 is compared to obtain the maximum probability corresponding to pixel position 1, and the category corresponding to the maximum probability is used to determine the category of pixel position 1. Similarly, to determine the category of pixel position 2 in the feature map, pointer Ptr0 needs to point to address 2, and the feature data needs to be read. Figure 1 The feature data at pixel position 2 is read, and then pointer Ptr0 jumps to address 102 to read the feature data. Figure 2 The feature data at pixel position 2 is then read, and pointer Ptr0 jumps to address 202 to read the feature. Figure 3 The feature data of pixel position 2 is finally used to read the features. Figure 1 Feature data and features of pixel position 2 Figure 2 Feature data and features of pixel position 2 Figure 3 The feature data of pixel position 2 is compared to obtain the maximum probability corresponding to pixel position 2, and the category corresponding to the maximum probability is determined to be the category of pixel position 2. However, in order to obtain the maximum probability corresponding to all pixel positions, the above-mentioned process of skipping through the data multiple times must be executed. Therefore, it can be seen that in this scheme, the number of skipping through the data is large and the time delay is long when calculating the maximum probability of each pixel position.

[0059] Based on this, in this embodiment of the application, in order to reduce the number of read skips and shorten the latency when calculating the maximum probability at each pixel position (i.e., performing the argmax operation), a scheme is proposed to set separate pointers for each of the N feature maps. Each feature map corresponds to one pointer, and there will be N pointers for N feature maps. The pointers corresponding to the N feature maps are controlled to move synchronously across the N feature maps in memory, resulting in the feature data pointed to by each of the N pointers. The feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to that feature map. For example, features... Figure 1 The pointer points to the feature data, which includes: features Figure 1 The probability that each pixel position belongs to category 1, feature Figure 2 The pointer points to the feature data, which includes: features Figure 2 The probability that each pixel position in the feature map belongs to category 2 is given by the pointer in the feature map N. The feature data pointed to by the pointer in the feature map N includes the probability that each pixel position in the feature map N belongs to category N.

[0060] For example, each pointer points to the memory address of a feature map, and the N feature maps have the same resolution. For instance, combining the above example, the feature... Figure 1The corresponding pointer is pointer Ptr1, and pointer Ptr1 points to the characteristic Figure 1 The memory addresses include: address 1 to address 100. Features Figure 2 The corresponding pointer is pointer Ptr2, and pointer Ptr2 points to the characteristic Figure 2 The memory addresses include addresses 101 to 200. (Features) Figure 3 The corresponding pointer is pointer Ptr3, and pointer Ptr3 points to the characteristic Figure 3 The memory addresses include: address 201 to address 300.

[0061] In one possible implementation, step 203 above controls the pointers corresponding to the N feature maps to move synchronously in the N feature maps in memory, and the implementation of obtaining the feature data pointed to by the N pointers includes the following steps 2031 to 2032:

[0062] Step 2031: Control the pointers corresponding to the N feature maps to move synchronously in the memory addresses corresponding to the N feature maps.

[0063] Specifically, the image processing device can control N pointers to move synchronously, so that all N pointers can simultaneously point to the storage address corresponding to the target pixel position. The pixel coordinates corresponding to the target pixel positions in the N feature maps are the same.

[0064] N feature maps can establish pixel coordinate systems based on the same rules. For example, each of the N feature maps can establish a pixel coordinate system with its own top-left vertex as the origin. In specific implementations, they can also establish pixel coordinate systems with their own top-right vertex, bottom-left vertex, or bottom-right vertex as the origin. This embodiment does not impose any specific limitations on this. Since the N feature maps establish pixel coordinate systems based on the same rules, and the N feature maps have the same resolution, if the N pointers move to the storage address corresponding to the pixel position with pixel coordinates (x, y) in each of the N feature maps, it can be considered that the N pointers have moved to the storage address corresponding to the target pixel position in each of the N feature maps.

[0065] For example, if N pointers move to the memory address corresponding to the pixel position in the first row and first column of each of the N feature maps, then the memory address corresponding to the target pixel position in each of the N feature maps is determined. Similarly, if N pointers move to the memory address corresponding to the pixel position in the first row and second column of each of the N feature maps, then the memory address corresponding to the target pixel position in each of the N feature maps is also determined. In other words, if N pointers move to the memory address corresponding to the pixel position in the x-th row and y-th column of each of the N feature maps, then the memory address corresponding to the target pixel position in each of the N feature maps is also determined.

[0066] For example, taking the three pointers (Ptr1, Ptr2, Ptr3) corresponding to the three feature maps mentioned above as an example, controlling the synchronous movement of these three pointers can be understood as follows: when the control pointer Ptr1 moves to address 1, the control pointer Ptr2 moves to address 101, and the control pointer Ptr3 moves to address 201. Addresses 1, 101, and 201 all correspond to pixel coordinates (1,1), and pixel coordinates (1,1) represent the target pixel position 1 in the first row and first column. The pixel coordinates (1,1) corresponding to address 1 are the feature map coordinates. Figure 1 The pixel coordinates (1,1) in the image, the pixel coordinates (1,1) corresponding to address 101 are features. Figure 2 The pixel coordinates (1,1) in the image, and the pixel coordinates (1,1) corresponding to address 201 are features. Figure 3 The pixel coordinates in the image are (1,1). At this point, it is assumed that the three pointers have moved to the storage address corresponding to the target pixel position 1 in each of the three feature maps, and the pixel coordinates of this target pixel position 1 are (1,1). The feature data stored at address 1 includes the probability that the target pixel position 1 belongs to category 1; the feature data stored at address 101 includes the probability that the target pixel position 1 belongs to category 2; and the feature data stored at address 201 includes the probability that the target pixel position 1 belongs to category 3.

[0067] When control pointer Ptr1 moves to address 2, control pointer Ptr2 moves to address 102, and control pointer Ptr3 moves to address 202. Addresses 2, 102, and 202 all correspond to pixel coordinates (1,2), which represent the target pixel position 2 in the first row and second column. The pixel coordinates (1,2) corresponding to address 2 are features. Figure 1 The pixel coordinates (1,2) in the image, the pixel coordinates (1,2) corresponding to address 102 are features. Figure 2 The pixel coordinates (1,2) in the image, and the pixel coordinates (1,2) corresponding to address 202 are features. Figure 3 The pixel coordinates in the image are (1,2). At this point, we assume the three pointers have moved to the storage address corresponding to the target pixel position 2 in each of the three feature maps, where the pixel coordinates of target pixel position 2 are (1,2). The feature data stored at address 2 includes the probability that target pixel position 2 belongs to category 1; the feature data stored at address 102 includes the probability that target pixel position 2 belongs to category 2; and the feature data stored at address 202 includes the probability that target pixel position 2 belongs to category 3. The method for controlling the movement of the three pointers to other addresses can be referred to the example above; to avoid repetition, it will not be repeated here.

[0068] In one possible implementation, step 2031 above includes: controlling N pointers to move to their respective starting addresses; and controlling the N pointers to move synchronously from their respective starting addresses according to preset pointer movement directions and pointer movement offsets.

[0069] In this implementation, the starting addresses of the N pointers each correspond to the initial pixel positions of the N feature maps. The pixel coordinates corresponding to the initial pixel positions in the N feature maps are the same. The initial pixel positions can be selected from the pixel positions in the feature maps as needed. Assume that each pixel position in the feature map includes A rows and B columns of pixel positions, with each row and column corresponding to one pixel position. In a specific implementation, any one of the following can be chosen as the initial pixel position: the pixel position of the top-left vertex (the pixel position corresponding to the first row and first column), the pixel position of the top-right vertex (the pixel position corresponding to the first row and second column), the pixel position of the bottom-left vertex (the pixel position corresponding to the first row and first column), or the pixel position of the bottom-right vertex (the pixel position corresponding to the first row and second column). Alternatively, the pixel position of the middle point of the feature map can also be chosen. However, this embodiment does not specifically limit the relative position of the initial pixel position within the feature map.

[0070] Specifically, each feature map's pointer initially points to the starting address of that feature map in memory, which is the memory address of the initial pixel position. The initial pixel position can be the pixel position corresponding to the first row and first column of the feature map. Then, N pointers are moved to their respective starting addresses. The starting address to which each pointer moves stores the feature data of the pixel position in the first row and first column of the feature map corresponding to that pointer. For example, the starting address (address 0) to which pointer Ptr1 moves stores the feature data. Figure 1 The feature data of the pixel position in the first row and first column is stored at the starting address (address 101) to which pointer Ptr2 moves. Figure 2 The feature data of the pixel position in the first row and first column is stored at the starting address (address 201) to which pointer Ptr3 moves. Figure 3 Feature data of the pixel position in the first row and first column.

[0071] The pointer movement direction represents the direction of movement of each pointer during synchronous movement and can be preset according to actual needs. Specifically, the pointer movement direction can be horizontal or vertical. The pointer movement offset determines the distance the pointer moves in memory each time. If the feature map is stored in memory row by row, then when moving the pointer horizontally, the offset is equal to the size of a single pixel; when moving the pointer vertically, the offset is equal to the total size of a row of pixels.

[0072] After determining the pointer movement direction and pointer offset, control N pointers to move synchronously from their respective starting addresses according to the pointer movement direction and pointer offset, so that the relative positions (i.e. offsets relative to their respective starting addresses) pointed to by the N pointers in their respective feature maps are the same.

[0073] In this embodiment, by setting the starting address of N pointers to correspond to the same initial pixel position, it is ensured that the starting point for reading all feature map data is consistent. This helps to accurately synchronize the reading process of different feature maps and ensures the accuracy of subsequent data processing. The preset pointer movement direction and offset ensure that the N pointers move according to the same rules, guaranteeing the consistency of the data read from different feature maps at the pixel position. By setting explicit starting addresses and unified movement rules for the N pointers, the pointer control logic is simplified, and the possibility of errors is reduced.

[0074] Step 2032: As the N pointers move to the storage address corresponding to the target pixel position in each of the N feature maps, read the feature data from the storage address pointed to by the N pointers respectively.

[0075] In this system, the pixel coordinates corresponding to the target pixel positions in the N feature maps are the same, and the feature data includes the probability that the target pixel position in each feature map belongs to the category corresponding to that feature map. That is, when the N pointers move to the storage address corresponding to the same pixel coordinates in the N feature maps, the feature data in the storage address pointed to by each of the N pointers is read.

[0076] For example, referring to the example above, suppose N pointers include features Figure 1 The corresponding pointer Ptr1 and features Figure 2 The corresponding pointer Ptr2 and the feature Figure 3 The corresponding pointer Ptr3, characteristics Figure 1 The corresponding category is denoted as category 1, and the feature is... Figure 2 The corresponding category is denoted as category 2, and the feature is... Figure 3 The corresponding category is denoted as category 3. When these three pointers move to the memory address corresponding to the target pixel position (taking target pixel position 1 as an example), it indicates that the feature data in the memory address pointed to by pointer Ptr1 includes the feature... Figure 1 The probability that target pixel position 1 belongs to category 1, and the feature data in the memory address pointed to by pointer Ptr2 includes features. Figure 2 The probability that target pixel position 1 belongs to category 2, and the feature data in the memory address pointed to by pointer Ptr3 includes features. Figure 3The probability that target pixel position 1 belongs to category 3. When all three pointers move synchronously to the storage address corresponding to target pixel position 2, it indicates that the feature data in the storage address pointed to by pointer Ptr1 includes the feature... Figure 1 The probability that target pixel position 2 belongs to category 1, and the feature data in the memory address pointed to by pointer Ptr2 includes features. Figure 2 The probability that target pixel position 2 belongs to category 2, and the feature data in the memory address pointed to by pointer Ptr3 includes features. Figure 3 The probability that target pixel position 2 belongs to category 3 is calculated. Similarly, each time the three pointers move synchronously once, the probability that a target pixel position in one of the three feature maps belongs to one of the three categories can be read.

[0077] It should be noted that the above example is for illustrative purposes only, using N=3 as an example. In actual implementation, this is not a limitation, and the value of N can be any value related to the number of object categories in the image to be processed.

[0078] Based on the feature data in the memory addresses pointed to by N pointers, determine the semantic segmentation result of the first resolution corresponding to the image to be processed.

[0079] It is understandable that feature data includes the probability that the target pixel position in each feature map belongs to the category corresponding to that feature map. Therefore, the feature data in the storage address pointed to by N pointers actually includes: the probability that the target pixel position belongs to each of the N categories, that is, N probabilities. The category corresponding to the highest probability among the N probabilities is taken as the category corresponding to the target pixel position.

[0080] For example, referring to the above example, when N=3, the feature data in the storage addresses pointed to by the three pointers includes: the probability that the target pixel position belongs to category 1, the probability that it belongs to category 2, and the probability that it belongs to category 3. Assuming that the probability of belonging to category 3 is the highest, then the category corresponding to the target pixel position is determined to be category 3. Based on this, as the three pointers move synchronously, feature data in different storage addresses can be read, thereby determining the category corresponding to each pixel position in the feature map, and thus obtaining the semantic segmentation result of the first resolution corresponding to the image to be processed.

[0081] In this embodiment, an independent pointer is set for each feature map. This allows for continuous access to the feature data of each feature map through its corresponding pointer, eliminating the need for skipping between different feature maps. This reduces latency caused by skipping access and avoids invalid addressing caused by skipping access, thereby reducing power consumption. By setting an independent pointer for each feature map, data at the same pixel position in N feature maps can be read simultaneously as the N pointers move synchronously. This accelerates the execution speed of the argmax operation, enabling rapid determination of the category corresponding to the same pixel position, thereby further improving the computational efficiency in the post-processing of semantic segmentation.

[0082] As described above, the semantic segmentation result at the first resolution is actually a low-resolution semantic segmentation result. In order to obtain a high-resolution semantic segmentation result for a specific category, further processing can be performed based on the semantic segmentation result at the first resolution.

[0083] In an exemplary embodiment, after determining the semantic segmentation result of the first resolution corresponding to the image to be processed based on the feature data, the method further includes, for example,... Figure 3 Steps 301 to 304 are shown below:

[0084] Step 301: Obtain the mask image of the specified category based on the semantic segmentation result of the first resolution.

[0085] The specified category can be any one or more of the N categories mentioned above, and can be selected according to the actual segmentation requirements. For example, if a second-resolution semantic segmentation result is desired for the category of "white cloud" among the N categories, then "white cloud" can be selected as the specified category. The second resolution is greater than the first resolution, therefore the semantic segmentation result of the second resolution can be understood as a high-resolution semantic segmentation result.

[0086] Since the semantic segmentation result at the first resolution includes the category corresponding to each pixel position in the feature map, the mask image generated based on this for a specified category can be a binary mask image. In this mask image, the category corresponding to the pixel position marked as 1 is the specified category, and the category corresponding to the pixel position marked as 0 is not the specified category. That is, the pixel position marked as 1 belongs to the specified category, and the category corresponding to the pixel position marked as 0 does not belong to the specified category. For example, if the specified category is cloud, the mask image corresponding to this specified category can be found in [reference needed]. Figure 4 .

[0087] Step 302: Among the N feature maps, determine the feature map corresponding to the specified category.

[0088] Understandably, the semantic segmentation model outputs N feature maps corresponding to N categories, allowing the selection of a feature map corresponding to a specific category from these N feature maps. If the specified category is "white cloud," then the feature map corresponding to "white cloud" is selected from the N feature maps. The feature map corresponding to "white cloud" represents the probability that each pixel position in that feature map belongs to the category of "white cloud."

[0089] Step 303: Determine the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category.

[0090] In this sub-image, each pixel position can be roughly classified into a specified category. The resolution of the target feature image is higher than that of the feature image corresponding to the specified category, and the target feature image can be obtained by cropping the feature image corresponding to the specified category based on the mask image.

[0091] In an exemplary embodiment, step 303 is implemented by the following steps S11 to S13:

[0092] S11: Upsample the mask image and the feature map corresponding to the specified category respectively to obtain the first upsampled image of the mask image and the second upsampled image of the feature map corresponding to the specified category.

[0093] In this model, the first and second upsampled images have the same resolution, which is higher than the resolution of the feature map. This upsampling is to ensure that the resolution of the first and second upsampled images matches the resolution of the image to be processed in the input semantic segmentation model. For example, if the resolution of the image to be processed is 100×100, and the resolution of the feature map and mask image corresponding to the specified category is 10×10, then the resolution of the first and second upsampled images is 100×100.

[0094] S12: On the first upsampled image, determine the bounding box corresponding to the pixel position of the specified category.

[0095] Here, the outer bounding box can be understood as: the smallest bounding box in the first upsampled image that can enclose all pixel positions belonging to the specified category. The shape of the smallest bounding box can be a rectangle, a circle, a trapezoid, etc., and this embodiment does not specifically limit it. When the shape of the smallest bounding box is a rectangle, it can be understood as: the outer rectangle corresponding to the pixel position of the specified category.

[0096] For example, see Figure 5 , Figure 5 This is a schematic diagram of the bounding box. Assuming the specified category is cloud, the bounding box corresponding to the pixel position of the cloud in the first upsampled image 501 can be... Figure 5 The circumscribed rectangle in the array is 502.

[0097] S13: Based on the bounding box, extract the feature sub-image corresponding to the specified category from the second upsampled image, and use the feature sub-image corresponding to the specified category as the target feature sub-image.

[0098] Specifically, the relative position of the bounding box in the first upsampled image can be determined first. Based on this relative position, the bounding box is moved to the second upsampled image, where its relative position is the same as in the first upsampled image. The image in the second upsampled image that is within the bounding box is then used as the extracted feature sub-image corresponding to the specified category.

[0099] In this embodiment, the target feature sub-image is essentially a cropped image from the upsampled mask image (first upsampled image) and the upsampled feature map (second upsampled image) corresponding to the specified category. Since the upsampled image has higher image detail and richness, combining the first and second upsampled images helps improve the accuracy of the semantic segmentation result for the specified category, reducing jagged edges at the segmentation boundaries and making the segmentation result appear smoother and more natural. Furthermore, since the upsampling at this point only applies to the mask image and the feature map corresponding to the specified category, upsampling is not required for feature maps other than the specified category. Therefore, this upsampling operation will not significantly impact the computational efficiency in the post-processing of semantic segmentation.

[0100] In an exemplary embodiment, step 303 is implemented by the following steps S21 to S23:

[0101] S21: On the mask image, determine the bounding box corresponding to the pixel position of the specified category.

[0102] The bounding box determined in this step is similar to the bounding box determined in S12 above, except that: in S12 above, the bounding box was determined on the upsampled mask image (the first upsampled image), while in this step, the bounding box is determined directly on the mask image. The bounding box in S21 is: the smallest bounding box in the mask image that can enclose all pixel positions belonging to the specified category.

[0103] S22: Based on the bounding box, extract the feature sub-map corresponding to the specified category from the feature map corresponding to the specified category.

[0104] Specifically, the relative position of the bounding box in the mask image can be determined first. Based on this relative position, the bounding box is moved to the feature map corresponding to the specified category. The relative position of the bounding box in the feature map corresponding to the specified category is the same as its relative position in the mask image. The image within the bounding box in the feature map corresponding to the specified category is then used as the extracted feature sub-map corresponding to the specified category.

[0105] S23: Upsample the feature sub-image to obtain a third upsampled image, and use the third upsampled image as the target feature sub-image.

[0106] Since neither the mask image nor the feature map corresponding to the specified category was upsampled in S21 and S22 above, the extracted feature sub-map corresponding to the specified category is still a low-resolution feature sub-map. Therefore, this step upsamples the feature sub-map to obtain a third upsampled image, which is then used as the target feature sub-map. This ensures that the target feature sub-map is a high-resolution feature sub-map compared to the mask image and the feature map corresponding to the specified category, thus facilitating the subsequent acquisition of high-resolution semantic segmentation results.

[0107] Specifically, the resolution ratio between the original image to be processed and the feature map can be used as the upsampling ratio to upsample the feature sub-image. For example, if the resolution of the original image to be processed is 100×100 and the resolution of the feature map output by the semantic segmentation model is 10×10, the resolution ratio between the image to be processed and the feature map is (100×100) / (10×10). Assuming the resolution of the feature sub-image is 2×2, then the feature sub-image is upsampled according to the sampling ratio of (100×100) / (10×10), resulting in a third upsampled image with a resolution of 20×20.

[0108] In this embodiment, the target feature sub-image is essentially a cropped feature map of the mask image corresponding to the upsampled feature map of the specified category, and then further upsampled. Since only the cropped feature sub-image needs to be upsampled, instead of the full-scale feature map, the computational load of upsampling is greatly reduced. Furthermore, the third upsampled image can reflect richer details related to the specified category, thus improving the accuracy of the semantic segmentation result at the second resolution corresponding to the specified category. Additionally, upsampling reduces jagged edges on the segmentation boundaries, making the segmentation result appear smoother and more natural.

[0109] Step 304: Based on the target feature sub-map, determine the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0110] The second resolution is greater than the first resolution.

[0111] Since the target feature map is a high-resolution feature map compared to the feature map output by the semantic segmentation model, a high-resolution semantic segmentation result corresponding to a specified category can be determined based on this target feature map. As mentioned earlier, each pixel position in the target feature map can be roughly classified into the specified category. Based on this, in order to perform more precise category classification on each pixel position in the target feature map, foreground and background can be further divided based on the target feature map to obtain a high-resolution semantic segmentation result for the specified category.

[0112] In this embodiment, based on actual needs, only the mask image of a specified category can be obtained based on the semantic segmentation result at the first resolution. Further processing can then be performed on this specified category, without needing to process all categories, which helps reduce computational load. Further processing based on the mask image and feature map of the specified category yields a target feature sub-map. This target feature sub-map integrates the feature information from the mask image and the feature map of the specified category, ensuring that the semantic segmentation result contains both global information and local details. This enhances the comprehensiveness and accuracy of the segmentation result, resulting in higher precision for the semantic segmentation result obtained for the specified category.

[0113] In an exemplary embodiment, step 304 above, which determines the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the target feature sub-map, includes the following steps 3041 to 3042:

[0114] Step 3041: Divide the target feature sub-image into foreground and background to obtain the foreground region in the target feature sub-image.

[0115] Specifically, a preset automatic thresholding segmentation algorithm can be used to divide the target feature sub-image into foreground and background to obtain the foreground region in the target feature sub-image. The preset automatic thresholding segmentation algorithm can be any one of the following algorithms: Otsu's Method, Triangle Method, and Adaptive Thresholding.

[0116] For example, the implementation process of step 3041 may include the following steps S31 to S33:

[0117] S31: Determine the gray-level histogram of the target feature sub-map. This gray-level histogram shows the frequency of each gray level in the image, which is crucial for determining the threshold.

[0118] S32: Based on the selected automatic thresholding algorithm, analyze the grayscale histogram and calculate the optimal global or local threshold. For example, in Otsu's method, the algorithm searches for an optimal threshold that maximizes the inter-class variance between the foreground and background.

[0119] S33: Based on the calculated optimal threshold, the pixels in the target feature sub-image are divided into two categories: pixels with gray values ​​higher than the optimal threshold are marked as foreground, and pixels with gray values ​​lower than or equal to the optimal threshold are marked as background, thus completing the division between foreground and background and obtaining the foreground region in the target feature sub-image.

[0120] In this embodiment, the application of the automatic threshold segmentation algorithm is beneficial for refining the boundaries of the foreground region, which helps to accurately separate the foreground object from the background, thereby obtaining the precise pixel position of the foreground part.

[0121] Step 3042: Based on the foreground region in the target feature sub-map, determine the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0122] Since the target feature sub-image is a cropped image, its coordinate system and size differ from those of the image to be processed. Therefore, the foreground region of the target feature sub-image can be mapped onto the image to be processed. In other words, the foreground region in the target feature sub-image is restored to the original image to be processed, thereby obtaining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0123] In an exemplary embodiment, step 3042 is implemented as follows: S41 to S42:

[0124] S41: Map the foreground region in the target feature sub-map to the image to be processed to obtain the mapped region of the foreground region in the image to be processed.

[0125] Specifically, the coordinate mapping relationship between the target feature sub-image and the image to be processed can be determined first. This coordinate mapping relationship describes the relationship between the pixel coordinates in the target feature sub-image and the pixel coordinates in the image to be processed. Then, based on this coordinate mapping relationship, the coordinates of the foreground region in the target feature sub-image in the image to be processed are determined, thus obtaining the mapped region of the foreground region in the image to be processed.

[0126] S42: Determine the mapped region as the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0127] In other words, the pixels in the mapped region of the image to be processed belong to the specified category. Specifically, the pixels in the mapped region can be labeled with the specified category, and the pixels in the image to be processed labeled with the specified category are determined as the semantic segmentation result of the second resolution corresponding to the specified category.

[0128] In some embodiments, a schematic diagram of the image processing method can be found in [reference needed]. Figure 6 ,include:

[0129] First, the image to be processed is input into the semantic segmentation model, which outputs N low-resolution feature maps.

[0130] Next, these N low-resolution feature maps are copied into memory. The argmax operation is then performed based on these copied feature maps to obtain the low-resolution semantic segmentation result. During the argmax operation, N pointers are used to point to the storage addresses of different feature maps, and these N pointers are moved synchronously to reduce the number of memory jumps and shorten the feature data acquisition time. Figure 6 In the example of low-resolution semantic segmentation results including pixel positions belonging to category 1 and pixel positions belonging to category 2, the feature map corresponding to category 1 is the feature map. Figure 1 The feature map corresponding to category 2 is the feature map. Figure 2 .

[0131] Then, based on the low-resolution semantic segmentation results, a mask image corresponding to the specified category (taking category 1 as an example) is obtained, and the mask image and the feature map corresponding to the specified category are compared. Figure 6 Features Figure 1 Both are upsampled to obtain the first upsampled image of the mask image and the feature image. Figure 1 The second upsampled image.

[0132] Furthermore, on the first upsampled image, the circumscribed rectangle 601 corresponding to the pixel position of category 1 is determined. Based on the relative position of the circumscribed rectangle 601 in the first upsampled image, the circumscribed rectangle 601 is moved to the second upsampled image. The relative position of the circumscribed rectangle 601 in the second upsampled image is the same as its relative position in the first upsampled image.

[0133] Furthermore, the image within the bounding rectangle 601 is extracted from the second upsampled image as the target feature sub-image.

[0134] Finally, an automatic thresholding segmentation algorithm is used to obtain the pixel positions of the foreground portion in the target feature sub-image. These pixel positions are then mapped back to the image to be processed, yielding a high-resolution semantic segmentation result for the specified category. Mapping these pixel positions back to the image to be processed can be understood as mapping the foreground region in the target feature sub-image to the image to be processed, thus obtaining the mapped region of the foreground region in the image to be processed.

[0135] In this embodiment, the argmax operation is performed directly on the low-resolution feature map copied in memory, instead of on the high-resolution feature map, which greatly reduces the computational load. Furthermore, it eliminates the need to store the feature data of the high-resolution feature map in memory, further reducing memory usage. When obtaining the high-resolution semantic segmentation result, only the truncated feature sub-map needs to be upsampled, rather than the full-scale feature map. Only foreground pixels of the specified category within the bounding rectangle of the foreground portion are calculated, further reducing computational load. Simultaneously, using multiple pointers when performing the argmax operation on the low-resolution feature map reduces memory jumps and accelerates feature data retrieval.

[0136] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application.

[0137] For example, such as Figure 7 As shown, the image processing apparatus 700 includes: an acquisition module 701, used to acquire N feature maps of the image to be processed output by a semantic segmentation model; wherein N > 1, each feature map corresponds to a category, and the resolution of each of the N feature maps is less than a preset resolution threshold; a copying module 702, used to copy the N feature maps into memory; a control module 703, used to control the pointers corresponding to each of the N feature maps to move synchronously in the N feature maps in memory, to obtain feature data pointed to by each of the N pointers; wherein the feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to the feature map; and a determination module 704, used to determine the semantic segmentation result of the image to be processed at a first resolution based on the feature data; wherein the semantic segmentation result includes the category corresponding to each pixel position, and the first resolution is less than the preset resolution threshold.

[0138] In one possible implementation, each of the aforementioned pointers is used to point to a storage address of one of the aforementioned feature maps in the aforementioned memory; the aforementioned N feature maps have the same resolution; the control module 703 includes: a synchronous movement control unit, used to control the pointers corresponding to each of the aforementioned N feature maps to move synchronously; a reading unit, used to control the pointers corresponding to each of the aforementioned N feature maps to move synchronously in the storage addresses corresponding to the N feature maps in the aforementioned memory; the reading unit is used to read the feature data in the storage addresses pointed to by the N aforementioned pointers each time the N aforementioned pointers move to the storage address corresponding to the target pixel position of each of the aforementioned N feature maps; wherein, the pixel coordinates corresponding to the target pixel positions of the N aforementioned feature maps are the same, and the aforementioned feature data includes the probability that the target pixel position in each of the aforementioned feature maps belongs to the category corresponding to the aforementioned feature map.

[0139] In one possible implementation, the synchronous movement control unit is specifically used to: control N pointers to move to their respective starting addresses; wherein the starting addresses of the N pointers correspond to the initial pixel positions of the N feature maps, and the pixel coordinates corresponding to the initial pixel positions of the N feature maps are the same; and control the N pointers to move synchronously from their respective starting addresses according to a preset pointer movement direction and pointer movement offset.

[0140] In one possible implementation, the image processing apparatus 700 further includes: a mask image acquisition module, configured to acquire a mask image of a specified category based on the semantic segmentation result at the first resolution; a feature map determination module, configured to determine the feature map corresponding to the specified category among the N feature maps; a target feature sub-map determination module, configured to determine the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category; and a semantic segmentation result determination module, configured to determine the semantic segmentation result of the specified category at a second resolution in the image to be processed based on the target feature sub-map; wherein the second resolution is greater than the first resolution.

[0141] In one possible implementation, the target feature sub-image determination module is specifically used to: upsample the mask image and the feature map corresponding to the specified category respectively to obtain a first upsampled image of the mask image and a second upsampled image of the feature map corresponding to the specified category; determine the bounding box corresponding to the pixel position of the specified category on the first upsampled image; and based on the bounding box, extract the feature sub-image corresponding to the specified category from the second upsampled image and use the feature sub-image corresponding to the specified category as the target feature sub-image.

[0142] In one possible implementation, the target feature sub-image determination module is specifically used to: determine the bounding box corresponding to the pixel position of the specified category on the mask image; based on the bounding box, extract the feature sub-image corresponding to the specified category from the feature image corresponding to the specified category; upsample the feature sub-image to obtain a third upsampled image, and use the third upsampled image as the target feature sub-image.

[0143] In one possible implementation, the semantic segmentation result determination module includes: a foreground determination unit, used to divide the target feature sub-image into foreground and background to obtain a foreground region in the target feature sub-image; and a semantic segmentation result determination unit, used to determine the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the foreground region in the target feature sub-image.

[0144] In one possible implementation, the semantic segmentation result determination unit is specifically used to: map the foreground region in the target feature sub-image to the image to be processed, to obtain the mapped region of the foreground region in the image to be processed; and determine the mapped region as the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

[0145] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0146] For example, such as Figure 8 As shown, the electronic device 800 includes a memory 801 and a processor 802. The memory 801 stores executable program code 8011, and the processor 802 is used to call and execute the executable program code 8011 to perform an image processing method.

[0147] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform an image processing method provided in embodiments of this application.

[0148] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0149] When each functional module is divided according to its corresponding function, the device may further include an acquisition module, a copying module, a control module, a determination module, etc. It should be noted that all relevant content regarding the steps involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0150] It should be understood that the apparatus provided in this embodiment is used to perform the above-described image processing method, and therefore can achieve the same effect as the above-described implementation method.

[0151] When using integrated units, the device may include a processing module and a storage module. When applied to an electronic device, the processing module can be used to control and manage the operation of the electronic device. The storage module can be used to support the execution of relevant program code by the electronic device.

[0152] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0153] In addition, the apparatus provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute an image processing method provided in the above embodiments.

[0154] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement an image processing method provided in the above embodiment.

[0155] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement an image processing method provided in the above embodiment.

[0156] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0157] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0158] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain N feature maps of the image to be processed from the output of the semantic segmentation model; where N > 1, each feature map corresponds to a category, and the resolution of each of the N feature maps is less than a preset resolution threshold. Copy the N feature maps into memory; The pointers corresponding to the N feature maps are controlled to move synchronously in the N feature maps in memory to obtain the feature data pointed to by the N pointers respectively; wherein, the feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to the feature map; Based on the feature data, a semantic segmentation result at a first resolution corresponding to the image to be processed is determined; wherein, the semantic segmentation result includes the category corresponding to each pixel position, and the first resolution is less than the preset resolution threshold.

2. The method according to claim 1, characterized in that, Each of the pointers is used to point to the storage address of one of the feature maps in the memory, and the N feature maps have the same resolution; The control of the pointers corresponding to the N feature maps moves synchronously across the N feature maps in memory to obtain the feature data pointed to by each of the N pointers, including: Control the pointers corresponding to each of the N feature maps to move synchronously within the storage addresses corresponding to the N feature maps in the memory; Each time the N pointers move to the storage address corresponding to the target pixel position in the N feature maps, feature data is read from the storage address pointed to by the N pointers respectively; wherein the pixel coordinates corresponding to the target pixel positions in the N feature maps are the same, and the feature data includes the probability that the target pixel position in each feature map belongs to the category corresponding to the feature map.

3. The method according to claim 2, characterized in that, The step of controlling the pointers corresponding to the N feature maps to move synchronously within the storage addresses corresponding to the N feature maps in memory includes: Control N pointers to move to their respective starting addresses; wherein, the starting addresses of the N pointers correspond to the initial pixel positions of the N feature maps, and the pixel coordinates corresponding to the initial pixel positions of the N feature maps are the same; According to the preset pointer movement direction and pointer movement offset, control N pointers to move synchronously from their respective starting addresses.

4. The method according to claim 1, characterized in that, After determining the semantic segmentation result of the first resolution corresponding to the image to be processed based on the feature data, the method further includes: Based on the semantic segmentation result at the first resolution, obtain a mask image of the specified category; From the N feature maps, determine the feature map corresponding to the specified category; Based on the mask image of the specified category and the feature map corresponding to the specified category, determine the target feature sub-map corresponding to the specified category; Based on the target feature sub-map, a semantic segmentation result with a second resolution corresponding to the specified category in the image to be processed is determined; wherein the second resolution is greater than the first resolution.

5. The method according to claim 4, characterized in that, The step of determining the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category includes: The mask image and the feature map corresponding to the specified category are upsampled respectively to obtain a first upsampled image of the mask image and a second upsampled image of the feature map corresponding to the specified category; On the first upsampled image, determine the bounding box corresponding to the pixel position of the specified category; Based on the bounding box, a feature sub-image corresponding to the specified category is extracted from the second upsampled image, and the feature sub-image corresponding to the specified category is used as the target feature sub-image.

6. The method according to claim 4, characterized in that, The step of determining the target feature sub-map corresponding to the specified category based on the mask image of the specified category and the feature map corresponding to the specified category includes: On the mask image, determine the bounding box corresponding to the pixel position of the specified category; Based on the bounding box, extract the feature sub-image corresponding to the specified category from the feature image corresponding to the specified category; The feature sub-image is upsampled to obtain a third upsampled image, and the third upsampled image is used as the target feature sub-image.

7. The method according to claim 4, characterized in that, The step of determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the target feature sub-map includes: The target feature sub-image is divided into foreground and background to obtain the foreground region in the target feature sub-image; Based on the foreground region in the target feature sub-map, determine the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

8. The method according to claim 7, characterized in that, The step of determining the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed based on the foreground region in the target feature sub-image includes: The foreground region in the target feature sub-image is mapped onto the image to be processed to obtain the mapped region of the foreground region in the image to be processed; The mapped region is determined as the semantic segmentation result of the second resolution corresponding to the specified category in the image to be processed.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire N feature maps of the image to be processed output by the semantic segmentation model; wherein, N>1, each feature map corresponds to a category, and the resolution of the N feature maps is less than a preset resolution threshold. The copy module is used to copy the N feature maps into memory; A control module is used to control the pointers corresponding to the N feature maps to move synchronously in the N feature maps in memory, so as to obtain the feature data pointed to by the N pointers respectively; wherein, the feature data includes the probability that each pixel position in each feature map belongs to the category corresponding to the feature map; The determining module is used to determine the semantic segmentation result of the first resolution corresponding to the image to be processed based on the feature data; wherein the semantic segmentation result includes the category corresponding to each pixel position, and the first resolution is less than the preset resolution threshold.

10. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 8.

Images