Image processing method and device, electronic equipment and storage medium

Abstract

This application provides an image processing method, apparatus, electronic device, and storage medium. Relating to the field of image processing, this application solves the problems of real-time performance and accuracy in pose estimation of target images. The specific solution involves: acquiring an RGB image and an infrared image including the target object; fusing the RGB image and the infrared image to obtain a fused image; segmenting the fused image to obtain the target object; and segmenting the target object to obtain a pose estimation result. The embodiments of this application are used for processing RGB images and infrared images including the target object.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of technology, robots are playing a vital role in various scenarios, including daily life, industry, and the military. To enable robots to function like humans, research is needed in many areas. Specifically, research in vision is crucial, as robots rely on visual perception to recognize objects and complete subsequent tasks.

[0003] In autonomous robots, the translation and orientation estimation of the objects being manipulated will be 6 degrees of freedom, or 6D pose. 6D pose estimation provides the robot with the object's position relative to itself in a spatial coordinate system. As 6D pose estimation technology continues to improve, both its accuracy and speed are increasing. However, the challenge of balancing high accuracy with real-time estimation remains to be addressed. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, electronic device, and storage medium that acquires RGB and infrared images of a target object in real time and performs fusion and segmentation processing on the images to solve the accuracy problem of pose estimation of the target object.

[0005] In a first aspect, embodiments of this application provide an image processing method, which includes: acquiring an RGB image and an infrared image including a target object; fusing the RGB image and the infrared image to obtain a fused image; segmenting the fused image to obtain the target object; and performing pose estimation on the target object to obtain a pose estimation result.

[0006] The above image processing method acquires the RGB and infrared images of the target object in real time, first fuses the RGB and infrared images to solve the problem of object occlusion, then segments the fused image to separate the target object from the surrounding background, and finally estimates the pose of the target object to obtain high-precision pose estimation results.

[0007] In conjunction with the first aspect, optionally, acquiring RGB and infrared images of the target object includes: spatial and temporal synchronization of the visible light camera and the infrared camera; acquiring RGB images based on the visible light camera; and acquiring infrared images based on the infrared camera.

[0008] The image processing method described above can simultaneously acquire RGB and infrared images of the target object by spatially and temporally synchronizing visible light and infrared cameras. RGB images offer advantages such as high resolution and rich scene information, while infrared images have strong detection and recognition capabilities and are less affected by external environmental factors. Therefore, combining the complementary advantages of RGB and infrared images can improve the accuracy of pose estimation results.

[0009] In conjunction with the first aspect, optionally, the process of fusing the RGB image and the infrared image to obtain a fused image includes: extracting features from the RGB image based on a convolutional neural network to obtain a first feature image; extracting features from the infrared image based on a convolutional neural network to obtain a second feature image; fusing the first feature image and the second feature image to obtain a third feature image; and extracting features from the third feature image based on a convolutional neural network to obtain the fused image.

[0010] The image processing method described above fuses RGB and infrared images to obtain a fused image. This fused image combines the complementary advantages of both RGB and infrared images, resulting in a bright target and a rich background. Furthermore, the fused image can resolve object occlusion issues.

[0011] In conjunction with the first aspect, optionally, segmenting the fused image to obtain the target object includes: convolving the fused image based on a convolutional neural network to obtain multiple convolution results; and superimposing the multiple convolution results to obtain the target object.

[0012] The image processing method described above can separate the target object from the surrounding background by segmenting the fused image, thus solving the problem of secondary information interfering with the important parts of the fused image.

[0013] In conjunction with the first aspect, optionally, the multiple convolution results include sequentially obtained first-layer convolution results, second-layer convolution results, third-layer convolution results, and fourth-layer convolution results. The target object is obtained by superimposing the multiple convolution results, including: superimposing the fourth-layer convolution result and the third-layer convolution result to obtain a fourth feature image; superimposing the fourth feature image and the second-layer convolution result to obtain a fifth feature image; superimposing the fifth feature image and the first-layer convolution result to obtain a sixth feature image; and performing a fully connected operation on the sixth feature image based on a convolutional neural network to obtain the target object.

[0014] The image processing method described above can improve image resolution by stacking the output of each layer with the upsampling results of each feature layer.

[0015] In conjunction with the first aspect, optionally, the process of performing pose estimation on the target pixel to obtain a pose estimation result includes: obtaining multiple feature points of the target object, and extracting feature points from the multiple feature points of the target object to obtain information about the multiple feature points of the target object, wherein the information about the feature points includes position information, orientation information, and thermal radiation intensity information; obtaining multiple feature vectors based on the information about the feature points; calculating and sorting the Euclidean distance between the multiple feature vectors to obtain a feature point matching result; and performing pose estimation on the feature point matching result based on the PNP pose estimation algorithm to obtain a pose estimation result.

[0016] The image processing method described above can improve the accuracy of pose estimation by fusing thermal radiation intensity information of the image and obtaining feature points based on the density and sparsity of thermal radiation intensity of the target object.

[0017] In conjunction with the first aspect, optionally, the process of obtaining multiple feature vectors based on the information of feature points includes: encoding the information of feature points and mapping the encoded information of feature points based on average pooling operations to obtain feature vectors.

[0018] Secondly, embodiments of this application also provide an image processing apparatus, including an acquisition module for acquiring an RGB image and an infrared image including a target object; a fusion module for fusing the RGB image and the infrared image to obtain a fused image; a segmentation module for segmenting the fused image to obtain the target object; and an estimation module for estimating the pose of the target object to obtain a pose estimation result.

[0019] The image processing apparatus provided in the above embodiments has the same beneficial effects as the image processing method provided in the first aspect or any optional embodiment of the first aspect, and will not be described in detail here.

[0020] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method described above.

[0021] The above embodiments provide an electronic device that has the same beneficial effects as the image processing method provided by the first aspect or any optional embodiment of the first aspect, which will not be elaborated here.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described above.

[0023] The computer-readable storage medium provided in the above embodiments has the same beneficial effects as the image processing apparatus provided in the first aspect or any alternative embodiment of the first aspect, which will not be repeated here.

[0024] In summary, this application provides an image processing method, apparatus, electronic device, and storage medium that can simultaneously meet the requirements of high-precision estimation, fast real-time operation, overcoming complex environments, and ignoring occlusion. This image processing method makes full use of the advantages of visible light cameras and infrared cameras, and can still estimate the target object in complex environments (such as smoke, fire, explosion, severe weather, and low visibility environments). At the same time, it constructs a lightweight network architecture to accurately estimate the pose of the target object in real time. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram illustrating the image processing method provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram of the visible light and infrared image fusion network provided in an embodiment of this application;

[0029] Figure 4 A flowchart illustrating the enhanced infrared semantic segmentation network provided in this application embodiment;

[0030] Figure 5 A schematic diagram of the object pose prediction network provided in an embodiment of this application;

[0031] Figure 6 A schematic flowchart of the image processing method provided in the embodiments of this application;

[0032] Figure 7 This is a schematic diagram of the modules of the image processing apparatus provided in the embodiments of this application;

[0033] Figure 8 A schematic diagram of the modules of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0036] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0037] Currently, to address the issues of accuracy and real-time estimation in 6D object pose estimation, one existing technology utilizes thermal radiation information from infrared images for 6D object pose estimation. This pose estimation technique is based on radar point cloud data and a depth camera to estimate the object's pose. However, this technique requires extensive computation and is not practically feasible. Furthermore, in some cases, the depth camera cannot accurately estimate the pose of occluded objects.

[0038] In another existing technology, a monocular camera can be used to estimate the pose of objects. This research primarily focuses on methods for class-level monocular 6D pose estimation and shape retrieval using metric combining, as well as a general model-free 6-DOF object pose estimator. This method is based on visible light images for pose estimation, which significantly reduces computational cost and increases processing speed. However, due to the limitations of visible light image features, pose estimation is not possible in special environments or when there is occlusion, thus hindering the achievement of high-precision estimation.

[0039] Therefore, this application provides an image processing method that acquires RGB and infrared images of a target object in real time, first fuses the RGB and infrared images to solve the problem of object occlusion, then segments the fused image to separate the target object from the surrounding background, and finally performs pose estimation on the target object to obtain a high-precision pose estimation result.

[0040] To facilitate understanding of this embodiment, the electronic device that performs the image processing method disclosed in this application will first be described in detail.

[0041] like Figure 1The diagram shown is a block diagram of an electronic device. The electronic device 100 may include a memory 111, a memory controller 112, a processor 113, a peripheral interface 114, an input / output unit 115, and a display unit 116. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] The aforementioned memory 111, memory controller 112, processor 113, peripheral interface 114, input / output unit 115, and display unit 116 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0043] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the electronic device 100 as defined in any embodiment of this application can be applied to or implemented by the processor 113.

[0044] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0045] The peripheral interface 114 described above couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 can be implemented on a single chip. In other instances, they can be implemented on separate chips.

[0046] The input / output unit 115 described above is used to provide user input data. The input / output unit 115 may be, but is not limited to, a mouse and keyboard, etc.

[0047] The aforementioned display unit 116 provides an interactive interface (e.g., a user interface) between the electronic device 100 and the user, or displays image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display (LCD) or a touch display. If it is a touch display, it can be a capacitive touchscreen or a resistive touchscreen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more locations on the touch display and pass the sensed touch operations to the processor for calculation and processing.

[0048] The electronic device 100 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the image processing method is described in detail below through several embodiments.

[0049] Please see Figure 2 The illustrated embodiment of this application provides a flowchart of an image processing method. This image processing method includes the following steps.

[0050] Step S201: The image processing device acquires an RGB image and an infrared image of the target object.

[0051] For example, RGB images and infrared images can be acquired at the same time and location. Infrared images are obtained by measuring the heat radiated outwards by a target object. Infrared images and grayscale images have the same data format and are both single-channel images. Furthermore, RGB images are also called visible light images. RGB images are 3-channel images, so they need to be converted to single-channel images for subsequent fusion with infrared images.

[0052] Infrared images possess excellent target detection and recognition capabilities, and can avoid the influence of external environmental factors such as smoke, lighting, and rain. However, infrared images also have some shortcomings, such as low pixel resolution, poor contrast, and blurred background textures. In contrast, RGB images are consistent with human visual characteristics, have high resolution, and can reflect rich scene information, such as texture and detail. However, RGB images are easily affected by environmental factors, and under conditions of environmental interference, RGB images may fail to highlight target objects.

[0053] Therefore, in this embodiment of the application, both RGB and infrared images of the target object are acquired simultaneously. This allows for the combination of the complementary advantages of RGB and infrared images, resulting in an image with a bright target and a rich background.

[0054] Step S202: The image processing device fuses the RGB image and the infrared image to obtain a fused image.

[0055] For example, an image processing device can fuse RGB and infrared images based on a convolutional neural network. The fused image combines the complementary advantages of RGB and infrared images, resulting in a bright target and a rich background. Furthermore, the fused image can resolve object occlusion issues.

[0056] Step 203: The image processing device segments the fused image to obtain the target object.

[0057] For example, the fused image includes a target image and a surrounding background, and the surrounding background can affect the pose estimation result of the subsequent target image. Therefore, in this embodiment, the fused image can be segmented based on a convolutional neural network, that is, semantic segmentation processing can be performed on the fused image to separate the target object from the surrounding background, and then the edge pixels of the target object can be further extracted to achieve the effect of separating the target object from the background. By separating the key objects in the fused image from the background, the problem of secondary information interfering with the important parts of the fused image can be solved.

[0058] Step 204: The image processing device performs pose estimation on the target object and obtains the pose estimation result.

[0059] For example, an image processing device can perform pose estimation of a target object based on a convolutional neural network. The pose estimation method can include feature point-based methods, line and surface-based methods, and deep learning-based methods. Specifically, in a feature point-based method, feature points are first extracted from the image, and then the motion of the object relative to the camera is calculated by matching feature points from two adjacent frames. In a line and surface-based method, line and surface information on the object's surface is first extracted, and line and surface features from two adjacent frames are matched in the image to calculate the motion of the object relative to the camera. In a deep learning-based method, a deep neural network is used to learn the image end-to-end, directly outputting the object's three-dimensional pose.

[0060] Optionally, step 201 may include: the image processing device performing spatial and temporal synchronization of the visible light camera and the infrared camera, acquiring an RGB image based on the visible light camera, and acquiring an infrared image based on the infrared camera.

[0061] For example, based on the specific relative installation positions and initial positioning of the visible light camera and the infrared camera, the pose information of the visible light camera and the infrared camera in the scene is obtained respectively. This pose information includes the coordinates of the visible light camera and the coordinates of the infrared camera. Then, the coordinates of the visible light camera are transformed to the coordinates of the infrared camera through translation vectors and rotation matrices, thereby completing the spatial synchronization of the visible light camera and the infrared camera.

[0062] In addition, the visible light camera and the infrared camera are triggered simultaneously to acquire RGB images and infrared images respectively. Spatial synchronization between the visible light camera and the infrared camera is achieved by matching adjacent timestamps of the two cameras to adjacent frames.

[0063] Therefore, by synchronizing the visible light camera and the infrared camera spatially and temporally, it is possible to ensure that the two cameras acquire data simultaneously, so that the RGB images and infrared images can be matched later.

[0064] For example, convolutional neural networks (CNNs) are a type of feedforward neural networks (FNNs) that include convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. A CNN can include an input layer, multiple convolutional layers, and pooling layers, etc.

[0065] The convolutional neural network executing step S202 can also be called a visible light and infrared image fusion network, such as... Figure 3As shown, the network may include 5 convolutional layers, 4 max pooling layers, 2 SOCA self-attention modules, 1 ResNet residual network, 1 dilated convolutional layer, 1 average pooling layer, and 1 activation function.

[0066] Applied to Figure 3 The visible light and infrared image fusion network shown may include, in step S202:

[0067] Step S2021: The image processing device extracts features from the RGB image based on a convolutional neural network to obtain a first feature image.

[0068] The RGB image is typically 512x512x2 in size. After a 5x5 convolutional layer to adjust the channels, a 512x512x16 feature map is generated. Then, a 3x3 max-pooling layer with a stride of 2 further extracts features, generating a 256x256x32 feature map. At this point, a residual network (ResNet) is partially inserted into the RGB image to avoid the gradient vanishing problem as the network deepens. The feature map then passes through a SOCA self-attention module. The SOCA module selects the most crucial information for the current task objective from a large amount of data, outputting a 256x256x32 feature map. This is then passed through another 5x5 convolutional layer to further extract features, outputting another 256x256x32 feature map. Finally, a 3x3 max-pooling layer with a stride of 2 outputs a 256x256x64 feature map, which is the first feature image.

[0069] Step S2022: The image processing device processes the infrared image based on a convolutional neural network to obtain a second feature image.

[0070] For example, corresponding to the processing of the RGB image in step S2021, the image processing device also processes the infrared image based on a convolutional neural network. The input infrared image first passes through a 5x5 convolutional layer to adjust the channels, then through a 5x5 max pooling layer with a stride of 2, then through a SOCA self-attention module, then through another 5x5 convolutional layer to adjust the channels, and finally through a 5x5 max pooling layer with a stride of 2 to obtain the second feature image.

[0071] Step S2023: The image processing device fuses the first feature image and the second feature image to obtain the third feature image.

[0072] For example, the first feature image obtained in step S2021 and the second feature image obtained in step S2022 are fused to obtain the third feature image.

[0073] Step S2024: The image processing device extracts features from the third feature image based on a convolutional neural network to obtain a fused image.

[0074] For example, the third feature image is passed through a 5x5 convolutional layer to adjust the channels, and the output image is 256x256x128. Then, it is passed through a 3x3 dilated convolution, which can increase the receptive field and reduce the computational cost, and the output image is 128x128x128. Next, it is passed through a 5x5 average pooling to suppress overfitting, and the output image is 128x128x256. Finally, the Tanh activation function is introduced to make the loss function converge faster, and a fused image of 128x128x256 is obtained.

[0075] In this embodiment, correlation metrics are performed for content prediction and texture prediction, including two parts: content loss and texture loss. The content loss L for RGB and infrared images is... content The mean squared error method can be used, and the definition of content loss is as follows:

[0076]

[0077] Where n is the size of the dataset loaded at one time during the training of the convolutional neural network, i.e., batchsize. The fused image feature representation extracted by the convolutional neural network at layer l is identified. This represents the feature representation of the original image at layer l.

[0078] In addition, the texture loss of RGB and infrared images can be calculated using the Gray matrix. The definition of texture loss is as follows:

[0079]

[0080] in, This represents the number of channels in the l-th layer. This represents the number of pixels in the feature map of layer l. The Gray matrix represents the feature representation of the fused image at layer l. The Gray matrix represents the feature representation of the original image at layer l.

[0081] Therefore, the overall loss function of the fused image is a linear weighted sum of content loss and texture loss, that is:

[0082] L total =αL content +βL surface

[0083] It could also be:

[0084]

[0085] This network incorporates the SOCA attention mechanism to extract more image features, preserving more detail from either RGB or infrared images and resulting in a clearer fused image. While enhancing the robustness of the convolutional neural network, a loss function is used to guide model optimization, enabling the fusion of RGB and infrared images during iterative training of the deep convolutional network.

[0086] Optionally, the convolutional neural network executing step S203 can also be called an enhanced infrared semantic segmentation network (EISS-Net), such as... Figure 4 As shown, the network can include two stages: upsampling and downsampling. The upsampling stage includes three convolutional layers, three CBAM attention modules, and one fully connected layer. The downsampling stage includes four convolutional layers, one residual block, and one pooling layer. The purpose of this network is to perform semantic segmentation on the input image, separating the target object from the surrounding background, and then further extracting the edge pixels of the target object to achieve the effect of separating the target object from the surrounding background. A residual network and a hybrid attention mechanism are introduced into the enhanced infrared semantic segmentation network. The residual network does not add any additional parameters and is relatively easy to improve. The depth can be increased through end-to-end backpropagation training to improve accuracy. The gradient vanishing problem caused by the increase in depth is alleviated by its internal skip connection mechanism. The attention mechanism can improve the network's sensitivity, make reasonable use of visual information processing resources, and then focus on important and useful information.

[0087] Applied to Figure 4 The enhanced infrared semantic segmentation network shown in step S203 may include:

[0088] Step S2031: The image processing device performs convolution on the fused image based on a convolutional neural network to obtain multiple convolution results.

[0089] For example, the image processing device performing convolution on the fused image to obtain multiple convolution results is a downsampling process. In the downsampling process of the enhanced infrared semantic segmentation network, there are three convolutional layers, one residual block, and one pooling layer. The residual block consists of two 3x3 convolutional layers, one 1x1 convolutional layer, and a ReLU activation function. The fused image can obtain multiple convolution results during the downsampling process of the enhanced infrared semantic segmentation network.

[0090] Step S2032: The image processing device superimposes multiple convolution results to obtain the target object.

[0091] For example, the image processing device superimposes the results of multiple convolutions to obtain the target object, which is an upsampling process. The upsampling process aims to restore the original size of the fused image to prepare for subsequent prediction. In the upsampling process, the output results of multiple layers in the downsampling process are stacked with the upsampling results of each feature layer, thereby improving the image resolution in the upsampling process.

[0092] Optionally, multiple convolution results include the first convolution result, the second convolution result, the third convolution result, and the fourth convolution result obtained sequentially.

[0093] For example, the fused image obtained in step S2024 has a size of 128x128x256. After a convolution of size 3x3 with a stride of 2, the fused image yields a first convolution result of 64x64x128. This is followed by another convolution of size 3x3 with a stride of 2, resulting in the first convolutional layer F1 (32x32x64). This first convolutional layer result is input into a residual block for processing, yielding a result of 16x16x32. This result is then processed by a 3x3 convolutional layer to obtain the second convolutional layer F2 (8x8x16). The second convolutional layer is then processed by another 3x3 convolutional layer with a stride of 2 to obtain the third convolutional layer F3 (4x4x8). Finally, the third convolutional layer is processed by a 2x2 pooling layer with a stride of 2 to output the fourth convolutional layer F4 (2x2x8). This completes the downsampling process.

[0094] Optionally, step S2032 may include:

[0095] Step S20321: Superimpose the results of the fourth convolution layer and the third convolution layer to obtain the fourth feature image.

[0096] For example, the result of the fourth convolution layer is upsampled using bilinear interpolation to output a 4x4x8 feature map, which is then superimposed on the result of the third convolution layer to output a 4x4x16 feature map. This feature map is then passed through the CBAM attention module and a 3x3 convolution layer to adjust the channels, resulting in an 8x8x32 fourth feature image.

[0097] Step S20322: The image processing device superimposes the fourth feature image and the second convolution result to obtain the fifth feature image.

[0098] For example, the fourth feature image is superimposed on the result of the second convolution layer, and then the channels are adjusted by the CBAM attention module and the 3x3 convolution layer to obtain the fifth feature image.

[0099] Step S20323: The image processing device superimposes the fifth feature image and the first layer convolution result to obtain the sixth feature image.

[0100] For example, the fifth feature image and the result of the first convolution layer are superimposed, and then the channels are adjusted through the CBAM attention module and the 3x3 convolution layer to obtain the sixth feature image of 32x32x128.

[0101] Step S20324: The image processing device performs a full connection on the sixth feature image based on a convolutional neural network to obtain the target image.

[0102] For example, the image processing device unfolds the sixth feature image into a one-dimensional vector through a fully connected circuit and provides it as input to the classifier. The classifier then makes a prediction using the softmax function and outputs a 32x32x128 target object.

[0103] The loss function of the enhanced infrared semantic segmentation network adopts the form of cross-entropy to represent the distance between the true value and the predicted value, and is used for backpropagation. Its calculation method is as follows: Where y represents the true value of the category,

[0104] This represents the predicted value.

[0105] Optionally, the convolutional neural network executing step S204 can also be called an object pose prediction network, such as... Figure 5 As shown, the process of an object pose prediction network can include: (1) feature point selection; (2) feature extraction; (3) feature point matching; (4) reducing mismatches; and (5) PNP. Specifically, this includes a 3x3 convolutional layer, a 2x2 pooling layer, a fully connected layer, and a regression layer.

[0106] Optionally, step S204 may include:

[0107] Step S2041: The image processing device acquires multiple feature points of the target object and extracts feature points from the multiple feature points of the target object to obtain information about the multiple feature points of the target object. The information about the feature points includes position information, orientation information and thermal radiation intensity information.

[0108] For example, an image processing device can acquire a target object segmented by an enhanced infrared semantic segmentation network, obtaining multiple feature points of the target object, such as 300 feature points. The position and orientation of each feature point are then extracted sequentially. Taking a certain image feature point C... i(i-1,…,300) For example, i is an integer greater than 1, with feature point C iConstruct a circle with a radius of 0.2 centered at a feature point. Then compare the feature matrices T at all feature point positions within the circle, where the feature matrix is... The eigenvalues ​​α and β of the feature matrix T represent the gradients in the x and y directions, respectively. The gradients of different feature points within the circle can be larger in one direction and smaller in the other. A threshold A can then be set, which can be used to filter feature points of interest in this embodiment. In one example, assuming α > β, points satisfying α / β > A will be discarded. It should be noted that the eigenvalues ​​of the feature matrix have no obvious functional relationship with the number of feature points. A minimum threshold (e.g., 200) is set for the number of feature points extracted per frame. If the number of extracted feature points is less than this threshold, the threshold A is decreased by a scale of 0.004 until the number of extracted feature points exceeds the minimum threshold.

[0109] After obtaining multiple feature points of interest, these feature points are used as input to a feature point information extraction structure to obtain information about the multiple feature points. This information includes location information W. i Directional information F i and thermal radiation intensity information T Pi The thermal radiation intensity information is unique to the fused image. The feature point information extraction structure can include a convolutional layer, a 2x2 pooling layer, a fully connected layer, and a regression layer. The pooling layer continuously reduces the input data size through downsampling to reduce computation. The convolutional layer has a kernel dimension of 1x3, a stride of 1, padding of 0, and uses the ReLU activation function. The fully connected layer improves the robustness of the overall feature point information extraction result. The regression network has a depth of 3, with 3 neurons in the input layer, 5 and 3 neurons in the hidden layers, and 2 neurons in the output layer. The hidden layers use the ReLU function. Therefore, the feature point information extraction structure can obtain two feature information for a feature point: the location information feature W. Pi directional information feature F Pi .

[0110] Step S2042: The image processing device obtains multiple feature vectors based on the information of the feature points.

[0111] For example, W Pi F Pi And the thermal radiation intensity T at each point Pi Feature point C is obtained by correlation. i Feature information abstract set P i ,in, Based on feature information abstract set Multiple feature vectors were obtained

[0112] Step S2043: The image processing device calculates and sorts the Euclidean distances of multiple feature vectors to obtain the feature point matching results.

[0113] For example, after obtaining the feature vectors corresponding to the feature points, the Euclidean distances between the feature points are calculated, and then all Euclidean distances are sorted to obtain the similarity between two feature points. Specifically, the two feature points with the smallest Euclidean distance are used as matching points for matching, and the distance metric is described as follows: Let be the feature vector corresponding to the feature point in the target image t. Let t+1 be the feature vector corresponding to the feature point in the target image. Calculate the matching distance for each selected feature point, sort all the matching distances, and select the feature point with the smallest distance or that meets the threshold requirement as the feature point matching result.

[0114] Because fused image features have local characteristics such as rotation and scaling, feature point mismatches are widespread, leading to many unnecessary feature matches and affecting matching accuracy. To reduce feature point mismatches, this application proposes an improvement to the matching algorithm by narrowing the matching range of feature points. Due to the unique thermal radiation intensity information of the fused image, the image can be easily divided into the target object and the surrounding background. Based on the density and sparsity of the thermal radiation intensity of the target object and the feature points obtained above, the entire image to be estimated is divided into grid regions of varying sizes. Regions with higher thermal radiation intensity feature points are retained and denser areas are extracted, while the intervening regions are removed. The final number of feature points in the retained regions is N < 200. Finally, precise matching is performed based on these regions, and the feature point closest to the target object in the matching region is selected as the matching point.

[0115] Step S2044: Based on the PNP pose estimation algorithm, the feature point matching results are used to estimate the pose, and the pose estimation results are obtained.

[0116] For example, by feature point C i Matching result pixel coordinates (X) i ,Y i The average length (l) and width (w) of the target object in the dataset correspond to the feature vector. A set of 3D pose prediction bounding box proposals for a target object is obtained, and this set is

[0117] Obtain the set of 3D pose prediction bounding boxes of the target object. First, calculate the area of ​​the smallest closure region between the two bounding boxes (the area of ​​the smallest box that includes both the predicted and ground truth boxes). Then, calculate the 3D Intersection over Union (IoU). Next, calculate the proportion of the region within the closure region that does not belong to either box (U is the union). Finally, subtract this proportion from the IoU to obtain the GIoU. The Intersection over Union with the highest overlap between the predicted and ground truth boxes, i.e., the highest GIoU, is denoted as GIoU. h ,in, Where -1≤GIoU h <1, A C The area of ​​the smallest bounding rectangle of the two rectangles is used to obtain the X coordinates of the predicted pose center point of the object. k and Y k ,in, Loss function L GIoU =1-GIoU, and 0<L GIoU ≤2.

[0118] By utilizing the principle of similar triangles in camera imaging, the coordinates of the target object in the world coordinate system can be inferred. Therefore, the PNP pose estimation algorithm is adopted: Where P is a point in the world coordinate system, P' is the point in the pixel coordinate system corresponding to P in the image, M1 is the camera's internal parameter matrix, and M2 is the camera's external parameter matrix.

[0119] This allows us to obtain the projection matrix from the world coordinate system to the pixel coordinate system, which represents the 6D pose information of the target object. The pose estimation results of the target object were obtained.

[0120] Optionally, step S2042 may include: the image processing device encoding the information of the feature points and mapping the encoded feature point information based on the average pooling operation to obtain a feature vector.

[0121] For example, after abstracting a set of feature point information, an encoder can be used to encode position, orientation, and thermal radiation intensity information. This encoder can be a ResNet-18 feature encoder. After encoding the feature point information, average pooling is used to map the information of different feature points into corresponding feature vectors.

[0122] Therefore, the flow of the image processing method provided in this application embodiment is as follows: Figure 6As shown, the RGB image and infrared image are synchronized in time and space, and then fused through an image fusion network (i.e., a visible light and infrared image fusion network) to obtain a fused image. The image is then segmented through an enhanced infrared semantic segmentation network to obtain the target object. Finally, the object pose prediction network is used to estimate the pose to obtain the 6D pose of the target object.

[0123] Please see Figure 6 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this application. An embodiment of this application provides an image processing apparatus 200, including: an acquisition module 210, a fusion module 220, a segmentation module 230, and an estimation module 240.

[0124] The acquisition module 210 is used to acquire RGB images and infrared images of the target object.

[0125] The fusion module 220 is used to fuse RGB images and infrared images to obtain a fused image.

[0126] The segmentation module 230 is used to segment the fused image to obtain the target object.

[0127] The estimation module 240 is used to perform pose estimation on the target object and obtain the pose estimation result.

[0128] It should be understood that this device corresponds to the image processing method embodiments described above and is capable of performing the various steps involved in the above method embodiments. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0129] Please see Figure 8 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0130] This application also provides a storage medium, which includes a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to perform the methods described above.

[0131] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0132] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0134] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire RGB and infrared images of the target object; The first feature image is obtained by extracting features from the RGB image using a convolutional neural network. Based on the convolutional neural network, feature extraction is performed on the infrared image to obtain a second feature image; The first feature image and the second feature image are fused to obtain the third feature image; Based on the convolutional neural network, feature extraction is performed on the third feature image to obtain a fused image; The fused image is convolved using a convolutional neural network to obtain multiple convolution results; The target object is obtained by superimposing the multiple convolution results; Multiple feature points of the target object are obtained, and feature point extraction is performed on the multiple feature points of the target object to obtain information of the multiple feature points of the target object. The information of the feature points includes position information, orientation information and thermal radiation intensity information. Based on the information of the feature points, multiple feature vectors are obtained; Calculate the Euclidean distance between the multiple feature vectors and sort them to obtain the feature point matching results; The pose estimation result is obtained by performing pose estimation on the feature point matching result based on the PNP pose estimation algorithm.

2. The method according to claim 1, characterized in that, The acquisition of the RGB and infrared images of the target object includes: Spatial and temporal synchronization of visible light and infrared cameras; The RGB image is acquired using the visible light camera, and the infrared image is acquired using the infrared camera.

3. The method according to claim 1, characterized in that, The multiple convolution results include a first convolutional layer result, a second convolutional layer result, a third convolutional layer result, and a fourth convolutional layer result obtained sequentially. The step of superimposing the multiple convolutional results to obtain the target object includes: The fourth convolutional result and the third convolutional result are superimposed to obtain the fourth feature image; The fourth feature image and the second convolution result are superimposed to obtain the fifth feature image; The fifth feature image and the first layer convolution result are superimposed to obtain the sixth feature image; The target object is obtained by performing a fully connected operation on the sixth feature image based on the convolutional neural network.

4. The method according to claim 1, characterized in that, Based on the information from the feature points, multiple feature vectors are obtained, including: The information of the feature points is encoded, and the encoded feature point information is mapped based on the average pooling operation to obtain multiple feature vectors.

5. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire RGB images and infrared images of the target object. The fusion module is used to extract features from the RGB image based on a convolutional neural network to obtain a first feature image; Based on the convolutional neural network, feature extraction is performed on the infrared image to obtain a second feature image; The first feature image and the second feature image are fused to obtain the third feature image; Based on the convolutional neural network, feature extraction is performed on the third feature image to obtain a fused image; The segmentation module is used to perform convolution on the fused image based on a convolutional neural network to obtain multiple convolution results; The target object is obtained by superimposing the multiple convolution results; An estimation module is used to obtain multiple feature points of the target object, and to extract feature points from the multiple feature points of the target object to obtain information about the multiple feature points of the target object. The information about the feature points includes position information, orientation information, and thermal radiation intensity information. Based on the information of the feature points, multiple feature vectors are obtained; Calculate the Euclidean distance between the multiple feature vectors and sort them to obtain the feature point matching results; The pose estimation result is obtained by performing pose estimation on the feature point matching result based on the PNP pose estimation algorithm.

6. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium includes a computer-readable storage medium; the computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 4.

Images