Image processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an electric power construction image, extracting an initial feature map from the electric power construction image, performing multi-scale coding based on the initial feature map, obtaining a global semantic feature map corresponding to the electric power construction image, performing multi-scale decoding on the global semantic feature map, obtaining a decoded feature map, and outputting the decoded feature map. Semantic segmentation is carried out on the electric power construction image based on the decoded feature map, a segmentation result corresponding to the electric power construction image is obtained, and the segmentation result comprises an image area in the electric power construction image and a category to which the image area belongs. By adopting the method, the accuracy of image semantic segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] As an important industry for national development and construction, electricity production and transportation maintain the normal operation of people's production and life. Power engineering is highly systematic and dynamic, resulting in a complex construction site environment. Power construction monitoring is of great significance in ensuring construction safety, improving construction quality, and managing construction progress and costs.

[0003] Therefore, image processing in power construction monitoring scenarios requires a certain degree of efficiency, and puts forward higher requirements on image fast encoding and decoding technology. However, traditional neural network methods are limited by their limited receptive field when processing high-resolution images, which affects the ability to semantically segment images. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product that can improve segmentation accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Acquire an electric power construction image, and extract an initial feature map from the electric power construction image;

[0007] Performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image;

[0008] Performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map;

[0009] The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0010] In a second aspect, the present application further provides an image processing device, comprising:

[0011] An acquisition module, used for acquiring an electric power construction image and extracting an initial feature map from the electric power construction image;

[0012] An encoding module, used for performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image;

[0013] A decoding module, used for performing multi-scale decoding on the global semantic feature map to obtain a decoding feature map;

[0014] A segmentation module is used to perform semantic segmentation on the electric power construction image based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Acquire an electric power construction image, and extract an initial feature map from the electric power construction image;

[0017] Performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image;

[0018] Performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map;

[0019] The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] Acquire an electric power construction image, and extract an initial feature map from the electric power construction image;

[0022] Performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image;

[0023] Performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map;

[0024] The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0025] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0026] Acquire an electric power construction image, and extract an initial feature map from the electric power construction image;

[0027] Performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image;

[0028] Performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map;

[0029] The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0030] The above-mentioned image processing method, device, computer equipment, storage medium and computer program product obtain an electric power construction image, extract an initial feature map from the electric power construction image, perform multi-scale encoding based on the initial feature map, and obtain a global semantic feature map corresponding to the electric power construction image to extract semantic features of different scales. The global semantic feature map is decoded at multiple scales so that the feature map obtained by decoding contains more detailed semantic features. The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result of the electric power construction image, which includes the image area in the electric power construction image and the category to which the image area belongs, thereby improving the accuracy of semantic segmentation in the electric power construction scenario through multi-scale encoding and decoding processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0033] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0034] Figure 3 is a schematic diagram of the architecture of an encoder in one embodiment;

[0035] Figure 4 is a schematic diagram of the architecture of an encoder in another embodiment;

[0036] Figure 5 is a structural block diagram of an image processing device in one embodiment;

[0037] Figure 6is an internal structure diagram of a server in one embodiment;

[0038] Figure 7 FIG. 4 is a diagram showing the internal structure of a terminal in an embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the power construction image and sends the power construction image to the server 104. The server 104 extracts the initial feature map of the power construction image, performs multi-scale encoding based on the initial feature map, and obtains the global semantic feature map corresponding to the power construction image. The server 104 performs multi-scale decoding on the global semantic feature map to obtain the decoded feature map. The server 104 performs semantic segmentation on the power construction image based on the decoded feature map to obtain the segmentation result corresponding to the power construction image, and the segmentation result includes the image area in the power construction image and the category to which the image area belongs. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0041] In an exemplary embodiment, Figure 2 As shown, an image processing method is provided, which is applied to a computer device (the computer device may be as shown in FIG. Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0042] Step S202: acquiring an electric power construction image and extracting an initial feature map from the electric power construction image.

[0043] Specifically, the computer device obtains the power construction image fed back by the power construction shooting device, divides the power construction image into multiple image regions, performs convolution processing on each image region, and obtains a regional feature map corresponding to each image region. An initial feature map corresponding to the power construction image is generated according to the multiple regional feature maps.

[0044] In this embodiment, the regional feature map can be represented by a regional feature representation, and the initial feature map can be represented by an image feature representation. The regional feature representation can be a regional feature vector, and the image feature representation can be an image feature matrix composed of multiple regional feature vectors.

[0045] Step S204: Perform multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image.

[0046] Specifically, the computer device may perform encoding processing at a scale on the initial feature map to extract a semantic feature map at a scale from the initial feature map to obtain the semantic feature map at the scale.

[0047] The computer device performs encoding processing on the semantic feature map at the scale at another scale to obtain a semantic feature map at another scale. According to similar processing, after performing multi-scale encoding processing, the semantic feature map at the last scale is used as the global semantic feature map.

[0048] In this embodiment, performing multi-scale encoding based on the initial feature map refers to performing multiple downsampling processes based on the initial feature map to obtain semantic feature maps at various scales.

[0049] Step S206, performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map.

[0050] Specifically, the computer device performs a decoding process at a certain scale on the global semantic feature map to obtain a decoding feature map at the scale.

[0051] The computer device performs a decoding process at another scale on the decoding feature map at the scale to obtain a decoding feature map at another scale. According to a similar process, after performing a decoding process at the last scale, a decoding feature map at the last scale is obtained.

[0052] In this embodiment, the multiple scales of encoding are the same as the multiple scales of decoding. For example, if encoding at four scales is performed, decoding at four scales will also be performed.

[0053] Step S208, semantic segmentation is performed on the electric power construction image based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, where the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0054] Specifically, the computer device performs semantic segmentation on the power construction image based on the decoded feature map to segment each image area in the power construction image and identify the category to which each image area belongs in the power construction scenario.

[0055] In one embodiment, the power construction image is processed by an image processing model. The image processing model includes an encoder and a decoder. The power construction image is input into the encoder of the image processing model, and the encoder performs convolution processing on the power construction image to obtain an initial feature map. The encoder performs multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image. The structure of the encoder can be as follows: Figure 3 As shown, the encoder includes a layered convolution block and multiple encoding blocks. The layered convolution block repositions, scales and rotates the input power construction image by utilizing an adaptively deformable spatial transformer, so that the image processing model can be robust to different transformations of the input image and can automatically align features of different angles, sizes and positions.

[0056] In this embodiment, the layer structure convolution block can also fuse the feature map obtained after the spatial transformation with the initial feature map to retain the low-level features in the initial feature map. The layer structure convolution block can specifically be a Stem (Spatial Transformer Embedding) layer.

[0057] The feature map output by the layered convolutional block is used as the input of the encoding block. Different encoding blocks are used to extract semantic features of different scales. The semantic feature map output by the previous encoding block is used as the input of the next encoding block. Through the encoding processing of multiple encoding blocks, the feature map is down-sampled, achieving efficient feature extraction.

[0058] In the above-mentioned image processing method, by acquiring the power construction image, the initial feature map of the power construction image is extracted, and multi-scale encoding is performed based on the initial feature map, so that encoding can be performed at different levels of the feature map to extract semantic features and context information of different scales, so that local detail features and global features can be considered at the same time, thereby extracting richer and more powerful feature representations, and obtaining a global semantic feature map that integrates local features and global features. The global semantic feature map is decoded at multiple scales, so that the feature map obtained by decoding contains more detailed semantic features. Semantic segmentation is performed on the power construction image based on the decoded feature map to obtain the segmentation result of the power construction image, which includes the image area in the power construction image and the category to which the image area belongs, so that the accuracy of semantic segmentation in the power construction scenario can be improved through multi-scale encoding and decoding processing.

[0059] In one embodiment, extracting an initial feature map from a power construction image includes:

[0060] The electric power construction image is divided into a plurality of image regions, and a regional feature map of each image region is extracted respectively; and an initial feature map corresponding to the electric power construction image is determined according to the regional feature map of each image region;

[0061] Based on the initial feature map, multi-scale encoding is performed to obtain the global semantic feature map corresponding to the power construction image, including:

[0062] The position of each image region in the electric power construction image is determined, and the position embedding vector of each position is determined; multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the electric power construction image.

[0063] Specifically, the computer device divides the electric power construction image into a plurality of image regions, extracts a regional feature map of each image region, and generates an initial feature map corresponding to the electric power construction image according to the regional feature map of each image region.

[0064] The computer device determines the position of each image area in the power construction image and determines the position embedding vector of each position. Multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the power construction image.

[0065] In this embodiment, the power construction image is divided into multiple image regions, and the regional feature map of each image region is extracted respectively. According to the regional feature map of each image region, the initial feature map corresponding to the power construction image is determined, the position of each image region in the power construction image is determined, and the position embedding vector of each position is determined. Multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the power construction image, so that semantic features of different scales can be accurately extracted.

[0066] In one embodiment, the initial feature map is represented by an image feature matrix, and the global semantic feature map is represented by a global semantic feature matrix; multi-scale encoding is performed based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image, including:

[0067] For each scale in the multi-scale, the evolution parameter and the first discrete parameter are obtained. Based on the evolution parameter and the first discrete parameter, the image feature representation at the targeted scale is mapped to the state space to obtain the discrete representation vector corresponding to the targeted scale in the state space. Based on the second discrete vector and the discrete representation vector corresponding to the targeted scale in the state space, the global semantic feature representation corresponding to the power construction image at the targeted scale is determined.

[0068] Specifically, the initial feature map is represented by an image feature matrix, and the global semantic feature map is represented by a global semantic feature matrix. The computer device obtains the evolution parameter, the first discrete parameter, and the second discrete parameter corresponding to the state space, and maps the image feature representation at the targeted scale to the state space based on the evolution parameter and the first discrete parameter, and obtains the discrete representation vector corresponding to the targeted scale in the state space.

[0069] The computer device determines the global semantic feature representation corresponding to the power construction image at the targeted scale based on the second discrete vector and the corresponding discrete representation vector in the state space at the targeted scale.

[0070] In this embodiment, the computer device obtains the first state conversion formula and the second state conversion formula in the state space, maps the image feature representation at the targeted scale to the state space through the first state conversion formula, and obtains a discrete representation vector. The global semantic feature representation is determined based on the discrete representation vector through the second conversion formula. For example:

[0071] h t =Ah t-1 +Bx t

[0072] y t =C T h t

[0073] Among them, parameter A represents the evolution parameter, parameter B represents the first discrete parameter, and parameter C represents the second discrete parameter. t represents the input sequence, h t-1 represents the potential state, h t represents the update of the potential state, y t Represents the predicted output sequence.

[0074] In this embodiment, for each scale in the multi-scale, the evolution parameter and the first discrete parameter are obtained, and based on the evolution parameter and the first discrete parameter, the image feature representation at the targeted scale is mapped to the state space, and the corresponding discrete representation vector in the state space at the targeted scale is obtained, so that the continuous image feature representation can be mapped to a discrete feature representation. Based on the second discrete vector and the corresponding discrete representation vector in the state space at the targeted scale, the global semantic feature representation corresponding to the power construction image at the targeted scale is determined, so that more semantic information in the power construction image can be extracted.

[0075] In one embodiment, in multi-scale encoding, the resolution of the global semantic feature map obtained by encoding at a later scale is smaller than the resolution of the global semantic feature map obtained by encoding at a previous scale, and the encoding at the later scale is to continue encoding the global semantic feature map obtained by encoding at the previous scale;

[0076] In multi-scale decoding, the resolution of the decoding feature map obtained by decoding at a later scale is greater than the resolution of the decoding feature map obtained by decoding at a previous scale, and the decoding at the later scale is to continue decoding the decoding feature map obtained by decoding at the previous scale.

[0077] Specifically, the computer device performs convolution processing on the power construction image to obtain an initial feature map. The computer device can perform encoding processing on the initial feature map at one scale to obtain a global semantic feature map at the scale. The computer device performs encoding processing on the global semantic feature map at the scale at another scale to obtain a global semantic feature map at another scale. Moreover, the resolution of the global semantic feature map obtained by the encoding processing at the latter scale is smaller than the resolution of the global semantic feature map obtained by the encoding at the previous scale. Through multi-scale encoding, the resolution of the semantic feature map can be gradually reduced during the encoding process to reduce the amount of calculation.

[0078] After obtaining the global semantic feature map of the last scale, the global semantic feature map of the scale is decoded to obtain a decoded feature map at the scale. The decoded feature map at the scale is decoded at another scale to obtain a decoded feature map at another scale, until the decoded feature map of the last scale is obtained. In addition, in the multi-scale decoding process, the resolution of the decoded feature map obtained by decoding at the latter scale is greater than the resolution of the decoded feature map obtained by decoding at the previous scale, so that the resolution of the feature map is gradually restored during the decoding process.

[0079] In this embodiment, in multi-scale encoding, the encoding of the latter scale is to continue encoding the global semantic feature map obtained by the encoding of the previous scale, and the resolution of the global semantic feature map obtained by the encoding of the latter scale is smaller than the resolution of the global semantic feature map obtained by the encoding of the previous scale, so that the resolution of the semantic feature map can be gradually reduced during the multi-scale encoding process to reduce the amount of calculation, thereby improving the efficiency of feature extraction. In multi-scale decoding, the decoding of the latter scale is to continue decoding the decoding feature map obtained by the decoding of the previous scale, and the resolution of the decoding feature map obtained by the decoding of the latter scale is greater than the resolution of the decoding feature map obtained by the decoding of the previous scale, so that the resolution of the feature map can be gradually restored during the multi-scale decoding process, so that the decoded feature map contains more detailed semantic information, thereby improving the accuracy of semantic segmentation.

[0080] In one embodiment, Figure 4 As shown in the figure, a schematic diagram of the encoder architecture of an image processing model is provided. This architecture is not only closely related to Transformers, but also integrates the high efficiency of SSMs. The core of the model is its application of the state space dual model (SSD). Through the structured state space dual (SSD) framework, it achieves language model performance comparable to Transformer, while having the advantage of linear scaling in sequence length.

[0081] The encoder uses the SSD layer, which gradually reduces the dimension of the input image from H×W×3 to This achieves efficient feature extraction. This process involves linear transformation of the input sequence, where the transformation at each stage exploits the properties of structured matrices to optimize computational efficiency and resource usage.

[0082] The SSD algorithm is based on block decomposition of semi-separable matrices. This algorithm not only has linear complexity during training, but also shows efficient performance during inference. Compared with traditional models, it can significantly reduce the required computing resources and time when processing large data sets, while maintaining the accuracy and generalization ability of the model.

[0083] In addition, the model is designed with optimizations for modern hardware in mind, supporting tensor parallelism and sequence parallelism, which makes the model more efficient when trained on a large scale. The model's multi-head mode design, including multi-query, multi-key, and multi-value attention mechanisms, further improves the model's ability to handle complex data.

[0084] Finally, through the processing of the UperNet decoder, the features extracted by the encoder can be gradually upsampled to restore high-resolution images, achieving fast and accurate image encoding and decoding, and meeting the high-efficiency requirements of power construction monitoring image processing.

[0085] The core state space dual model (SSD) of the Mamba block can be represented by a linear ordinary differential equation with evolution parameters A, projection parameters B and C:

[0086] h t =Ah t-1 +Bx t

[0087] y t =C T h t

[0088] Among them, x t represents the input sequence, h t-1 represents the potential state, h t represents the update of the potential state, y t Represents the predicted output sequence.

[0089] SSD passes through the latent space h t x t Mapped to the response y t In deep learning models, the required state transitions are discrete rather than continuous. Therefore, it is important to discretize this state. Discrete outputs are obtained from sampled values ​​according to the time step of the input result.

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] in, and They represent the discretized matrices A and B respectively, and h represents the state of the time step.

[0096] In this embodiment, the layer convolution block is formed by combining a convolution layer and a projection layer (Linear Projection), and the output of the layer convolution block is normalized and used as the input to the Mamba block. The image feature representation extracted from the high-resolution power construction image is quite long, and the Mamba block models this image feature representation to extract semantic information from the context sequence. This method helps to effectively extract features from the power construction image.

[0097] In this embodiment, a combination of Mamba block and Multilayer Perceptron (MLP) is used in the coding block design to enhance the representation ability of the model and enhance the model's learning ability for complex data. Among them, the multilayer perceptron is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0098] In this embodiment, a fast encoding and decoding scheme for power construction monitoring images based on the Mamba large model is proposed. By combining the Mamba large model with the state space dual model, the fast encoding and decoding of high-resolution monitoring images is achieved through optimized state update equations and discretization strategies, which significantly improves the efficiency and accuracy of image processing. Due to the strong inductive ability of SSD in long sequences, the Mamba-based framework model shows excellent segmentation results compared with other methods, providing more complete and accurate segmentation of large terrain areas, and therefore performs well in minimizing false alarms. The integrity and accuracy of fast encoding and decoding of images are improved.

[0099] In order to improve computational efficiency, a state-space dual model is adopted. This state-space dual model is based on block decomposition of semi-separable matrices and makes full use of the matrix multiplication units of modern hardware accelerators, which significantly reduces the required computing resources while maintaining linear time complexity.

[0100] Moreover, the model supports tensor parallelism and sequence parallelism, which enables the model to process large-scale data sets more efficiently. In addition, the multi-head mode can be introduced to better capture the complex features in the image and improve the model's ability to learn information of different scales.

[0101] Due to the strong generalization ability of SSD in long sequences, the Mamba-based framework model shows excellent segmentation results compared with other methods, providing more complete and accurate segmentation of large terrain areas, thus performing well in minimizing false positives. The completeness and accuracy of fast encoding and decoding of images are improved.

[0102] In one embodiment, the segmentation result also includes the target object in the image region and the construction operation of the target object, and the method further includes:

[0103] Determine the operation matching degree between the construction operation of the target object and the preset construction operation; when the operation matching degree does not meet the preset operation matching threshold, mark the target object in the power construction image.

[0104] Specifically, the computer device performs semantic segmentation on the electric power construction image based on the decoded feature map to obtain an image area in the electric power construction image, as well as a target object in the image area and a construction operation of the target object.

[0105] The computer device can calculate the operation matching degree between the construction operation of the target object and the preset construction operation. When the operation matching degree does not meet the preset operation matching threshold, the target object is marked in the power construction image.

[0106] In this embodiment, the operation matching degree between the construction operation of the target object and the preset construction operation is determined to judge whether the operation of each target object in the power construction image complies with the power construction specification. When the operation matching degree does not meet the preset operation matching threshold, it means that the user's operation category does not comply with the preset power construction operation specification, and the user who does not comply with the operation specification is marked in the power construction image.

[0107] In one embodiment, the image frame in the electric power construction video captured by the electric power construction shooting device of the electric power construction image, the segmentation result also includes the construction safety tools of the target object, and the method further includes:

[0108] Determine the target object's wearing method for construction safety tools, and determine the wearing match between the wearing method and the preset wearing method of the preset safety tools; when the wearing match does not meet the preset wearing match conditions, send a prompt message to the target object through the power construction shooting equipment, and the prompt message is used to instruct the target object to adjust the wearing method of the construction safety tools.

[0109] Specifically, the image frames in the electric construction video shot by the electric construction shooting equipment of the electric construction image. In the electric construction scene, the electric construction shooting equipment can shoot within the electric construction range, and feed back the electric construction video obtained by shooting to the computer device. The computer device receives the electric construction video, extracts the electric video image from the electric construction video, extracts the initial feature map of the electric construction image, performs multi-scale encoding based on the initial feature map, obtains the global semantic feature map corresponding to the electric construction image, performs multi-scale decoding on the global semantic feature map, obtains the decoded feature map, performs semantic segmentation on the electric construction image based on the decoded feature map, and obtains the image area of ​​the electric construction image and the category to which the image area belongs, the target image in the image area, and the construction safety tools worn by the target object.

[0110] Determine the way in which the target object in the power construction image wears the construction safety tools, and determine the degree of matching between the target object's wearing method and the preset wearing method of the preset safety tools. When the wearing matching degree does not meet the preset wearing matching condition, a prompt message is sent to the target object through the power construction shooting device, and the prompt message is used to instruct the target object to adjust the wearing method of the construction safety tools.

[0111] In this embodiment, the computer device may be a monitoring device, and the electric power construction shooting device sends the shot electric power construction video to the monitoring device, and the monitoring device performs the above processing on the electric power construction video. When the wearing matching degree does not meet the preset wearing matching condition, the monitoring device prompts the electric power construction shooting device to send prompt information to the target object through the electric power construction shooting device to prompt the target object to adjust the wearing method of the construction safety tools.

[0112] In this embodiment, semantic segmentation is performed on the power construction image based on the decoded feature map to obtain each target object and the construction area where the target object is located in the power construction image. The priority of each construction area is determined based on the preset construction operation corresponding to the construction area. When the construction operation of the highest priority area has been completed, a remote control instruction is issued through the power construction shooting equipment. The remote control instruction is used to prompt the target object to move to the next construction area for power construction.

[0113] In one embodiment, an image processing method is provided, which is applied to a computer device and includes:

[0114] Acquire the electric power construction image, divide the electric power construction image into multiple image regions, and extract the regional feature map of each image region respectively.

[0115] According to the regional feature map of each image region, an initial feature map corresponding to the electric power construction image is determined.

[0116] For each scale in the multi-scale, an evolution parameter and a first discrete parameter are obtained, and based on the evolution parameter and the first discrete parameter, the image feature representation at the targeted scale is mapped to the state space to obtain a discrete representation vector corresponding to the targeted scale in the state space.

[0117] Based on the second discrete vector and the corresponding discrete representation vector in the state space at the targeted scale, a global semantic feature representation corresponding to the electric power construction image at the targeted scale is determined.

[0118] The global semantic feature map is decoded at multiple scales to obtain a decoded feature map.

[0119] Based on the decoded feature map, semantic segmentation is performed on the electric power construction image to obtain the segmentation result corresponding to the electric power construction image. The segmentation result includes the image area in the electric power construction image and the category to which the image area belongs, as well as the target object in the image area and the construction operation of the target object.

[0120] Optionally, the operation matching degree between the construction operation of the target object and the preset construction operation is determined, and when the operation matching degree does not meet the preset operation matching threshold, the target object is marked in the power construction image.

[0121] Optionally, the target object's wearing method for construction safety tools is determined, and the wearing matching degree between the wearing method and the preset wearing method of the preset safety tool is determined. When the wearing matching degree does not meet the preset wearing matching conditions, a prompt message is sent to the target object through the power construction shooting equipment, and the prompt message is used to instruct the target object to adjust the wearing method of the construction safety tools.

[0122] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0123] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0124] In an exemplary embodiment, Figure 5 As shown, an image processing device 500 is provided, comprising: an acquisition module 502, an encoding module 504, a decoding module 506 and a segmentation module 508, wherein:

[0125] The acquisition module 502 is used to acquire the electric power construction image and extract the initial feature map from the electric power construction image.

[0126] The encoding module 504 is used to perform multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image.

[0127] The decoding module 506 is used to perform multi-scale decoding on the global semantic feature map to obtain a decoded feature map.

[0128] The segmentation module 508 is used to perform semantic segmentation on the electric power construction image based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, where the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

[0129] In this embodiment, by acquiring the power construction image, extracting the initial feature map of the power construction image, and performing multi-scale encoding based on the initial feature map, encoding can be performed at different levels of the feature map to extract semantic features and context information of different scales, so that local features and global features can be taken into account at the same time, thereby extracting richer and more powerful feature representations, and obtaining a global semantic feature map that integrates local features and global features. The global semantic feature map is decoded at multiple scales, so that the feature map obtained by decoding contains more detailed semantic features. Semantic segmentation is performed on the power construction image based on the decoded feature map to obtain a segmentation result of the power construction image, which includes the image area in the power construction image and the category to which the image area belongs, so that the accuracy of semantic segmentation in the power construction scenario can be improved through multi-scale encoding and decoding processing.

[0130] In one embodiment, the acquisition module 502 is further used to divide the power construction image into multiple image regions, respectively extract the regional feature map of each image region; and determine the initial feature map corresponding to the power construction image according to the regional feature map of each image region;

[0131] The encoding module 504 is also used to determine the position of each image area in the power construction image and determine the position embedding vector of each position; multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the power construction image.

[0132] In this embodiment, the power construction image is divided into multiple image regions, and the regional feature map of each image region is extracted respectively. According to the regional feature map of each image region, the initial feature map corresponding to the power construction image is determined, the position of each image region in the power construction image is determined, and the position embedding vector of each position is determined. Multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the power construction image, so that semantic features of different scales can be accurately extracted.

[0133] In one embodiment, the initial feature map is represented by an image feature matrix, and the global semantic feature map is represented by a global semantic feature matrix; the encoding module 504 is also used to obtain an evolution parameter and a first discrete parameter for each scale in the multiple scales, and based on the evolution parameter and the first discrete parameter, map the image feature representation at the targeted scale to the state space to obtain a discrete representation vector corresponding to the targeted scale in the state space; based on the second discrete vector and the discrete representation vector corresponding to the targeted scale in the state space, determine the global semantic feature representation corresponding to the power construction image at the targeted scale.

[0134] In this embodiment, for each scale in the multi-scale, the evolution parameter and the first discrete parameter are obtained, and based on the evolution parameter and the first discrete parameter, the image feature representation at the targeted scale is mapped to the state space, and the corresponding discrete representation vector in the state space at the targeted scale is obtained, so that the continuous image feature representation can be mapped to a discrete feature representation. Based on the second discrete vector and the corresponding discrete representation vector in the state space at the targeted scale, the global semantic feature representation corresponding to the power construction image at the targeted scale is determined, so that more semantic information in the power construction image can be extracted.

[0135] In one embodiment, in multi-scale encoding, the resolution of the global semantic feature map obtained by encoding at a later scale is smaller than the resolution of the global semantic feature map obtained by encoding at a previous scale, and the encoding at the later scale is to continue encoding the global semantic feature map obtained by encoding at the previous scale;

[0136] In the multi-scale decoding process, the resolution of the decoding feature map obtained by decoding at the next scale is greater than the resolution of the decoding feature map obtained by decoding at the previous scale, and the decoding at the next scale is to continue decoding the decoding feature map obtained by decoding at the previous scale.

[0137] In this embodiment, in multi-scale encoding, the encoding of the latter scale is to continue encoding the global semantic feature map obtained by the encoding of the previous scale, and the resolution of the global semantic feature map obtained by the encoding of the latter scale is smaller than the resolution of the global semantic feature map obtained by the encoding of the previous scale, so that the resolution of the semantic feature map can be gradually reduced during the multi-scale encoding process to reduce the amount of calculation, thereby improving the efficiency of feature extraction. In multi-scale decoding, the decoding of the latter scale is to continue decoding the decoding feature map obtained by the decoding of the previous scale, and the resolution of the decoding feature map obtained by the decoding of the latter scale is greater than the resolution of the decoding feature map obtained by the decoding of the previous scale, so that the resolution of the feature map can be gradually restored during the multi-scale decoding process, so that the decoded feature map contains more detailed semantic information, thereby improving the accuracy of semantic segmentation.

[0138] In one embodiment, the segmentation result also includes the target object in the image region and the construction operation of the target object, and the apparatus further includes:

[0139] The marking module is used to determine the operation matching degree between the construction operation of the target object and the preset construction operation; when the operation matching degree does not meet the preset operation matching threshold, the target object is marked in the power construction image.

[0140] In this embodiment, the operation matching degree between the construction operation of the target object and the preset construction operation is determined to judge whether the operation of each target object in the power construction image complies with the power construction specification. When the operation matching degree does not meet the preset operation matching threshold, it means that the user's operation category does not comply with the preset power construction operation specification, and the user who does not comply with the operation specification is marked in the power construction image.

[0141] In one embodiment, the image frame in the electric power construction video captured by the electric power construction shooting device of the electric power construction image, the segmentation result also includes the construction safety tools of the target object, and the device also includes:

[0142] The marking module is used to determine the target object's wearing method for construction safety tools, and determine the wearing match between the wearing method and the preset wearing method of the preset safety tools; when the wearing match does not meet the preset wearing match conditions, a prompt is sent to the target object through a camera that shoots the power construction video, and the prompt is used to instruct the target object to adjust the wearing method of the construction safety tools.

[0143] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0144] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0145] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 6 , Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an electric power construction image, and extract an initial feature map from the electric power construction image; Performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image; Performing multi-scale decoding on the global semantic feature map to obtain a decoded feature map; The electric power construction image is semantically segmented based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

2. The method according to claim 1, characterized in that The extracting of an initial feature map from the electric power construction image comprises: Dividing the electric power construction image into a plurality of image regions, and extracting a regional feature map of each image region respectively; Determining an initial feature map corresponding to the power construction image according to the regional feature map of each image region; The multi-scale encoding is performed based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image, including: Determine the position of each of the image regions in the electric power construction image, and determine the position embedding vector of each position; Multi-scale encoding processing is performed based on the initial feature map and the position embedding vector to obtain a global semantic feature map corresponding to the power construction image.

3. The method according to claim 1, characterized in that The initial feature map is represented by an image feature matrix, and the global semantic feature map is represented by a global semantic feature matrix; the multi-scale encoding based on the initial feature map to obtain the global semantic feature map corresponding to the power construction image includes: For each scale in the multiple scales, an evolution parameter and a first discrete parameter are obtained, and based on the evolution parameter and the first discrete parameter, the image feature representation at the targeted scale is mapped to a state space to obtain a discrete representation vector corresponding to the targeted scale in the state space; Based on the second discrete vector and the corresponding discrete representation vector in the state space at the targeted scale, a global semantic feature representation corresponding to the power construction image at the targeted scale is determined.

4. The method according to claim 3, characterized in that In the multi-scale encoding, the resolution of the global semantic feature map obtained by encoding at a later scale is smaller than the resolution of the global semantic feature map obtained by encoding at a previous scale, and the encoding at the later scale is to continue encoding the global semantic feature map obtained by encoding at the previous scale; In the multi-scale decoding, the resolution of the decoding feature map obtained by decoding at a later scale is greater than the resolution of the decoding feature map obtained by decoding at a previous scale, and the decoding at the later scale is a continued decoding of the decoding feature map obtained by decoding at the previous scale.

5. The method according to any one of claims 1 to 4, characterized in that: The segmentation result also includes the target object in the image area and the construction operation of the target object, and the method further includes: Determining the degree of operation matching between the construction operation of the target object and the preset construction operation; When the operation matching degree does not meet a preset operation matching threshold, the target object is marked in the power construction image.

6. The method according to claim 5, characterized in that The image frame of the electric power construction video captured by the electric power construction shooting device of the electric power construction image, the segmentation result also includes the construction safety tool of the target object, and the method further includes: Determining a wearing mode of the target object for the construction safety tool, and determining a wearing matching degree between the wearing mode and a preset wearing mode of a preset safety tool; When the wearing matching degree does not meet the preset wearing matching condition, a prompt message is sent to the target object through the electric power construction shooting device, and the prompt message is used to instruct the target object to adjust the wearing method of the construction safety tool.

7. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an electric power construction image and extracting an initial feature map from the electric power construction image; An encoding module, used for performing multi-scale encoding based on the initial feature map to obtain a global semantic feature map corresponding to the power construction image; A decoding module, used for performing multi-scale decoding on the global semantic feature map to obtain a decoding feature map; A segmentation module is used to perform semantic segmentation on the electric power construction image based on the decoded feature map to obtain a segmentation result corresponding to the electric power construction image, wherein the segmentation result includes an image area in the electric power construction image and a category to which the image area belongs.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images