A three-dimensional reconstruction method, system, security inspection machine and storage medium

Abstract

The embodiment of the present application discloses a 3D reconstruction method, system, security inspection machine and storage medium, which relates to the field of computer vision technology. The method includes: determining the image to be processed according to the first monitoring image and the second monitoring image; determining the first intermediate feature sequence and the second intermediate feature sequence of the image to be processed; performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain the third intermediate feature sequence; inputting the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain the first target feature sequence output by the preset encoder; and completing the 3D reconstruction according to the target 3D point cloud data when the target 3D point cloud data is determined according to the second target feature sequence. In the entire 3D reconstruction process, there is no need to extract feature points, match feature points, triangulate point clouds and other implementation processes in the traditional 3D reconstruction method, thereby improving the efficiency of 3D reconstruction.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method, system, security inspection machine and storage medium. Background Art

[0002] 3D reconstruction technology refers to the establishment of a mathematical model suitable for computer representation and processing of 3D objects. It is the basis for processing, operating and analyzing the properties of 3D objects in a computer environment. It is also a key technology for establishing virtual reality in computers to express the objective world. 3D reconstruction technology is widely used in many fields such as computer graphics, robotics, medical imaging, geographic information systems, etc.

[0003] Traditional 3D reconstruction technology mainly includes sparse reconstruction and dense reconstruction. Among them, sparse reconstruction can calculate the 3D sparse point cloud of the scene based on the feature point matching sequence between the input images, and simultaneously estimate the intrinsic and extrinsic parameters of the camera. Dense reconstruction can calculate the dense spatial point cloud pixel by pixel based on the camera pose calculated by sparse reconstruction, which will take a long time to process.

[0004] It can be seen that traditional 3D reconstruction technology has the problem of low efficiency. Summary of the invention

[0005] In view of this, one of the objectives of the present application is to provide a three-dimensional reconstruction method, system, security inspection machine and storage medium, which can improve the efficiency of three-dimensional reconstruction.

[0006] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method, comprising:

[0008] Determine an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship;

[0009] Determine a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed;

[0010] Performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence;

[0011] Inputting the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder;

[0012] In the case where the target three-dimensional point cloud data is determined according to the second target feature sequence, three-dimensional reconstruction is completed according to the target three-dimensional point cloud data, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence.

[0013] In a possible implementation, determining a first intermediate feature sequence of an image to be processed includes:

[0014] Divide the image to be processed to obtain multiple sub-images to be processed;

[0015] Performing linear mapping processing on the multiple sub-images to be processed to determine an initial feature sequence, wherein the initial feature sequence includes multiple one-dimensional feature sequences obtained after performing linear mapping processing on the multiple sub-images to be processed;

[0016] Position encoding of the initial feature sequence;

[0017] A first intermediate feature sequence is determined according to the initial feature sequences before and after the position encoding.

[0018] In a possible implementation, determining the first intermediate feature sequence according to the initial feature sequences before and after the position encoding includes:

[0019] The initial feature sequences before and after position encoding are added together to obtain a first intermediate feature sequence.

[0020] In a possible implementation, determining a second intermediate feature sequence of the image to be processed includes:

[0021] Perform one-dimensional convolution of different dimensions on the image to be processed to obtain feature pyramids of different resolutions;

[0022] Input feature pyramids of different resolutions into a preset convolutional neural network to obtain feature sequences of different resolutions corresponding to the output of the preset convolutional neural network;

[0023] After performing this operation on feature sequences with different resolutions, a second intermediate feature sequence is obtained.

[0024] In a possible implementation, the second intermediate feature sequence includes a plurality of sub-feature sequences, and the first intermediate feature sequence and the second intermediate feature sequence are subjected to attention fusion processing to obtain a third intermediate feature sequence, including:

[0025] Determining the resolution of the first intermediate feature sequence and the resolution of each sub-feature sequence in the second intermediate feature sequence;

[0026] A third intermediate feature sequence is determined according to sub-feature sequences in the second intermediate feature sequence having the same resolution as that of the first intermediate feature sequence.

[0027] In a possible implementation, determining the third intermediate feature sequence according to a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence includes:

[0028] Adding the first sub-feature sequence to the first intermediate feature sequence to obtain a second sub-feature sequence;

[0029] Determine a target sequence according to the second sub-feature sequence and the third sub-feature sequence, wherein the first sub-feature sequence includes a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence, and the third sub-feature sequence is a sub-feature sequence in the second intermediate feature sequence excluding the first sub-feature sequence;

[0030] According to the preset encoder, determine the self-attention matrix of each sub-feature sequence in the target sequence;

[0031] When the resolution of the target sequence is determined, all self-attention matrices are interpolated according to the bilinear interpolation method to align the resolution of the target sequence with the resolution of the first intermediate feature sequence to obtain the third intermediate feature sequence.

[0032] In a possible implementation, the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence, including:

[0033] Performing attention fusion processing on the first target feature sequence and the second intermediate feature sequence to obtain a fourth intermediate feature sequence;

[0034] The fourth intermediate feature sequence, the second intermediate feature sequence and the first target feature sequence are added together to obtain a second target feature sequence.

[0035] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction system, the system comprising:

[0036] A first determining module is used to determine an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship;

[0037] A second determining module is used to determine a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed;

[0038] A first processing module is used to perform attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence;

[0039] A second processing module is used to input the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder;

[0040] The third processing module is used to complete the three-dimensional reconstruction according to the target three-dimensional point cloud data when the target three-dimensional point cloud data is determined according to the second target feature sequence, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence.

[0041] In a third aspect, an embodiment of the present application provides a security inspection machine, which includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the three-dimensional reconstruction method provided in the first aspect is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by one or more processors, it implements the three-dimensional reconstruction method provided in the first aspect.

[0043] The three-dimensional reconstruction method provided in the embodiment of the present application can determine the image to be processed based on the first monitoring image and the second monitoring image, and obtain the third intermediate feature sequence by performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence of the image to be processed, which can highlight the important features in the image to be processed and suppress unimportant or noise features. Then the first intermediate feature sequence and the third intermediate feature sequence are input into a preset encoder to obtain a first target feature sequence output by the preset encoder, and the target three-dimensional point cloud data is determined based on the second target feature sequence determined by the first target feature sequence and the second intermediate feature sequence to complete the three-dimensional reconstruction. In the entire three-dimensional reconstruction process, there is no need to extract feature points, match feature points, triangulate point clouds, and other implementation processes in the traditional three-dimensional reconstruction method, thereby improving the efficiency of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. It should be understood that the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0045] Figure 1 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of functional modules of a three-dimensional reconstruction system provided in an embodiment of the present application;

[0047] Figure 3 This is a diagram of the internal structure of a security inspection machine provided in an embodiment of the present application.

[0048] Description of reference numerals:

[0049] The three-dimensional reconstruction system 200 includes a first determination module 210 , a second determination module 220 , a first processing module 230 , a second processing module 240 , and a third processing module 250 . DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0053] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any combination or all combinations of the words listed at the same time. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0054] In the description of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0055] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0056] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0057] Furthermore, in the embodiments of the present application, the term "connection" may refer to "electrical connection" or "direct connection". "Electrical connection" may refer to direct electrical connection between two components or electrical connection between two components via one or more normally open tubes or other components.

[0058] In order to solve the technical problems in the background technology, the embodiments of the present application provide a three-dimensional reconstruction method, system, security inspection machine and storage medium. The three-dimensional reconstruction method provided by the embodiments of the present application is first introduced below.

[0059] See also Figure 1 , Figure 1 This is a flowchart of a 3D reconstruction method provided for an embodiment of the present application. The 3D reconstruction method can be applied to a security inspection machine, which is an important device for detecting the internal structure of luggage, parcels and other items to ensure safety. The security inspection machine in this embodiment can be used in airports, stations, subway stations and other places where security inspections are required. The 3D reconstruction method specifically includes the following steps:

[0060] Step 110, determining an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship;

[0061] Step 120, determining a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed;

[0062] Step 130, performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence;

[0063] Step 140, inputting the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder;

[0064] Step 150, completing three-dimensional reconstruction based on the target three-dimensional point cloud data when the target three-dimensional point cloud data is determined according to the second target feature sequence, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence.

[0065] The three-dimensional reconstruction method provided by the embodiment of the present application can determine the image to be processed based on the first monitoring image and the second monitoring image, and obtain the third intermediate feature sequence by performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence of the image to be processed, which can highlight the important features in the image to be processed and suppress unimportant or noise features. Then the first intermediate feature sequence and the third intermediate feature sequence are input into a preset encoder to obtain a first target feature sequence output by the preset encoder, and the target three-dimensional point cloud data is determined based on the second target feature sequence determined by the first target feature sequence and the second intermediate feature sequence to complete the three-dimensional reconstruction. In the entire three-dimensional reconstruction process, there is no need to extract feature points, match feature points, triangulate point clouds, and other implementation processes in the traditional three-dimensional reconstruction method, thereby improving the efficiency of three-dimensional reconstruction.

[0066] The following will be Figure 1 Each step of the method is described in detail.

[0067] In step 110, the first monitoring image and the second monitoring image may be images obtained by the security inspection machine in the X-ray scanning area. Specifically, the security inspection machine generally includes a conveyor belt, through which the items to be inspected can be conveyed into the X-ray scanning area of ​​the security inspection machine, and the first monitoring image and the second monitoring image can be obtained in the X-ray scanning area.

[0068] The security inspection machine in this embodiment includes security inspection equipment using X-ray binocular imaging technology, and the first monitoring image and the second monitoring image can be in a certain geometric distribution relationship. It should be noted that ordinary camera imaging only needs to capture the natural light reflected by the object, while X-ray imaging requires a specific X-ray light source and detection plate, and the image obtained by the X-ray binocular perspective requires two X-ray light sources and two detection plates.

[0069] In some embodiments, the first monitoring image and the second monitoring image may be in an orthogonal relationship. Specifically, the security inspection machine may perform a perspective scan on the items to be detected in the X-ray scanning area in a first scanning mode to obtain a first monitoring image, and perform a perspective scan on the items to be detected in the X-ray scanning area in a second scanning mode to obtain a second monitoring image. The first scanning mode includes scanning from top to bottom or from bottom to top. The second scanning mode includes scanning from left to right or from right to left.

[0070] In some embodiments, the security inspection machine can also obtain the second monitoring image through the first scanning method, and obtain the first monitoring image through the second scanning method.

[0071] Specifically, the security inspection machine can acquire the first monitoring image and the second monitoring image through the monitoring module provided inside the security inspection machine, such as an X-ray light source and two detection boards.

[0072] The security inspection machine may directly capture the first monitoring image and the second monitoring image through the monitoring module, or may determine the first monitoring image and the second monitoring image from the video stream collected by the monitoring module.

[0073] It should be noted that the first monitoring image and the second monitoring image may be acquired by two different monitoring modules in the security inspection machine at the same time.

[0074] The image to be processed can be an image obtained by the security inspection machine by aligning the first monitoring image and the second monitoring image in width and splicing them in the height direction. In fact, the security inspection machine fuses the first monitoring image and the second monitoring image in a conjoined manner to facilitate subsequent feature dimensionality reduction.

[0075] It should be noted that the conjoined feature dimensionality reduction can fully fuse the image features in the first monitoring image and the second monitoring image, providing rich contextual semantic information for each image region. If the first monitoring image and the second monitoring image are subjected to dimensionality reduction processing separately, the twin network needs to be used to perform dimensionality reduction and feature fusion on the first monitoring image and the second monitoring image respectively, which results in a complicated processing flow and low processing efficiency.

[0076] In step 120, the security inspection machine may perform different processing on the image to be processed, thereby obtaining a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed.

[0077] The first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed, wherein the multi-scale feature information includes high-dimensional semantic information and low-dimensional spatial information of the image to be processed.

[0078] The first intermediate feature sequence and the second intermediate feature sequence are both pixel sequences of the image to be processed.

[0079] In step 130, the attention mechanism enables the model to identify and focus on the most important parts of the input first intermediate feature sequence and the second intermediate feature sequence. The security inspection machine can integrate multi-receptive field space multi-scale convolution features into the first intermediate feature sequence by performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence, and obtain a third intermediate feature sequence. The subsequent steps are processed based on the third intermediate feature sequence, and the reliability and accuracy of the target three-dimensional point cloud data can be determined, thereby improving the reliability and accuracy of three-dimensional reconstruction.

[0080] In step 140, a preset encoder is used to process the first intermediate feature sequence and the third intermediate feature sequence, which can not only reduce redundant information in the first intermediate feature sequence and the third intermediate feature sequence while maintaining important features and improve processing efficiency, but also combine the first intermediate feature sequence and the third intermediate feature sequence for joint reasoning to obtain a rich and comprehensive feature, namely, the first target feature sequence.

[0081] In some embodiments, the preset encoder includes a ViT encoder, which is a Transformer-based visual encoder that can achieve joint reasoning on the first intermediate feature sequence and the third intermediate feature sequence, wherein the Transformer architecture is a neural network using a self-attention mechanism.

[0082] In step 150, after obtaining a target feature sequence in the above steps, the security inspection machine can determine the second target feature sequence through the first target feature sequence and the second intermediate feature sequence. The second target feature sequence determined by the security inspection machine is high-dimensional, and accurate target three-dimensional point cloud data can be determined through the second target feature sequence, which is also convenient for improving the accuracy and reliability of three-dimensional reconstruction. In addition, in the process of determining the target three-dimensional point cloud data by the security inspection machine, there is no need to extract feature points, match feature points, triangulate point clouds, etc. in the traditional three-dimensional reconstruction method, which can improve the efficiency of three-dimensional reconstruction.

[0083] In a possible implementation, determining a first intermediate feature sequence of an image to be processed includes:

[0084] Divide the image to be processed to obtain multiple sub-images to be processed;

[0085] Performing linear mapping processing on the multiple sub-images to be processed to determine an initial feature sequence, wherein the initial feature sequence includes multiple one-dimensional feature sequences obtained after performing linear mapping processing on the multiple sub-images to be processed;

[0086] Position encoding of the initial feature sequence;

[0087] A first intermediate feature sequence is determined according to the initial feature sequences before and after the position encoding.

[0088] This embodiment obtains a first intermediate feature sequence by dividing the image to be processed and performing linear mapping and position encoding operations on the multiple sub-images to be processed obtained by the division. This is beneficial for the subsequent first intermediate feature sequence to maintain the spatial consistency of features in the feature fusion process, thereby improving the quality and expression ability of the fused features, thereby improving the accuracy of determining the target three-dimensional point cloud data, and improving the accuracy and reliability of three-dimensional reconstruction.

[0089] Specifically, the image to be processed can be divided into equal parts, which can ensure that each sub-image to be processed obtained by the equal division has the same dimension after feature extraction, and is convenient for operations such as adding and splicing initial feature sequences before and after encoding to obtain a first intermediate feature sequence with high reliability.

[0090] Exemplarily, the image to be processed may be equally divided into 16*16 sub-images to be processed, with a total of 256 sub-images to be processed.

[0091] Multiple one-dimensional feature sequences can be concatenated in spatial order to obtain an initial feature sequence.

[0092] Linear mapping processing can achieve dimensionality reduction of features of multiple sub-images to be processed. It can not only remove redundant information in the image such as background noise, reduce the amount of data to be processed, and thus improve the processing efficiency of multiple sub-images to be processed, but also facilitate the input of the first intermediate feature sequence determined by operations such as linear mapping and position encoding into the ViT encoder for processing.

[0093] It is important to note that, unlike convolutional neural networks, the ViT encoder processes sequences rather than three-dimensional pixel matrices.

[0094] Among them, position encoding can retain the spatial information of each feature in the initial feature sequence, which is conducive to improving the accuracy of determining the target three-dimensional point cloud data, and then improving the accuracy and reliability of three-dimensional reconstruction.

[0095] In a possible implementation, determining the first intermediate feature sequence according to the initial feature sequences before and after the position encoding includes:

[0096] The initial feature sequences before and after position encoding are added together to obtain a first intermediate feature sequence.

[0097] By adding the initial feature sequences before and after position encoding, this embodiment can make the first intermediate feature sequence include not only the initial features in multiple sub-images to be processed, but also the spatial information of the initial features, which is conducive to determining accurate target three-dimensional point cloud data.

[0098] In a possible implementation, determining a second intermediate feature sequence of the image to be processed includes:

[0099] Perform one-dimensional convolution of different dimensions on the image to be processed to obtain feature pyramids of different resolutions;

[0100] Input feature pyramids of different resolutions into a preset convolutional neural network to obtain feature sequences of different resolutions corresponding to the output of the preset convolutional neural network;

[0101] The feature sequences with different resolutions are serially fused to obtain a second intermediate feature sequence.

[0102] This embodiment performs one-dimensional convolution processing of different dimensions on the processed image, and uses a preset convolutional neural network to process the feature pyramids of different resolutions obtained after the one-dimensional convolution processing. This can reduce the amount of calculation while ensuring the accuracy of feature extraction, improve the efficiency of determining the target three-dimensional point cloud data, and thus improve the efficiency of three-dimensional reconstruction.

[0103] Specifically, one-dimensional convolutions of different dimensions can be performed on the image to be processed according to different pre-set convolution kernel sizes and step sizes, so as to extract features of the image to be processed in different dimensions and obtain feature pyramids of different resolutions.

[0104] The preset convolutional neural network can realize depthwise separable convolution and linear mapping processing. Among them, depthwise separable convolution (DSC) is an algorithm obtained by improving the standard convolution calculation in convolutional neural networks (CNN), which can reduce the number of image features and the amount of calculation and improve processing efficiency. Linear mapping processing can map feature sequences of different resolutions to the same dimension through convolutional networks, which facilitates the series fusion of feature sequences of different resolutions.

[0105] Among them, serial fusion includes stacking feature sequences of different resolutions in order from small to large resolution.

[0106] In some embodiments, the preset convolutional neural network includes a deep separable convolutional neural network. Selecting the deep separable convolutional neural network can reduce the number of image features and the amount of calculation, and can achieve higher computing efficiency with the same computing resources.

[0107] Among them, information such as the convolution kernel size and step size can be set according to actual needs, and this embodiment does not limit this.

[0108] For example, feature pyramids of different resolutions can be used c 1 、c 2 、c 3 indicates that the feature sequences of different resolutions corresponding to the preset convolutional neural network output can be used Fc i It indicates that, i Indicates i The feature sequence of resolution. Fc 1, Fc 2 and Fc Example 3: The second intermediate feature sequence obtained by serial fusion F 2 can be simplified as (Fc 1 , Fc 2 ， Fc 3 ） .

[0109] Specifically, the second intermediate feature sequence can be calculated by the following formula: F 2, among which, .

[0110] , formula 1

[0111] , formula 2

[0112] in, FC represents a linear mapping, DWConv represents depthwise separable convolution, HW Indicates the resolution of the image to be processed. k n Indicates the downsampling ratio. d Represents dimension.

[0113] in, k 1 、k 2 、k 3 represents different downsampling ratios.

[0114] In a possible implementation, the second intermediate feature sequence includes a plurality of sub-feature sequences, and the first intermediate feature sequence and the second intermediate feature sequence are subjected to attention fusion processing to obtain a third intermediate feature sequence, including:

[0115] Determining the resolution of the first intermediate feature sequence and the resolution of each sub-feature sequence in the second intermediate feature sequence;

[0116] A third intermediate feature sequence is determined according to sub-feature sequences in the second intermediate feature sequence having the same resolution as that of the first intermediate feature sequence.

[0117] This embodiment determines the way of attention fusion of the first intermediate feature sequence and the second intermediate feature sequence through the resolution of the first intermediate feature sequence and the resolution of each sub-feature sequence in the second intermediate feature sequence, and obtains the third intermediate feature sequence. The third intermediate feature sequence can integrate the key features in the first intermediate feature sequence and the second intermediate feature sequence, and obtain the third intermediate feature sequence with richer feature expression while further reducing the calculation amount of feature data and improving calculation efficiency, so as to improve the accuracy and reliability of determining the target three-dimensional point cloud data.

[0118] As mentioned above, the second intermediate feature sequence is obtained by serially merging feature sequences of different resolutions, wherein the feature sequences of different resolutions are the multiple sub-feature sequences in this embodiment.

[0119] The resolution of the first intermediate feature sequence may be determined by the number of features in the first intermediate feature sequence.

[0120] In a possible implementation, determining the third intermediate feature sequence according to a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence includes:

[0121] Adding the first sub-feature sequence to the first intermediate feature sequence to obtain a second sub-feature sequence;

[0122] Determine a target sequence according to the second sub-feature sequence and the third sub-feature sequence, wherein the first sub-feature sequence includes a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence, and the third sub-feature sequence is a sub-feature sequence in the second intermediate feature sequence excluding the first sub-feature sequence;

[0123] According to the preset encoder, determine the self-attention matrix of each sub-feature sequence in the target sequence;

[0124] When the resolution of the target sequence is determined, all self-attention matrices are interpolated according to the bilinear interpolation method to align the resolution of the target sequence with the resolution of the first intermediate feature sequence to obtain the third intermediate feature sequence.

[0125] The preset encoder includes the ViT encoder introduced in the above embodiment. The target sequence is actually determined based on the first intermediate feature sequence and the second intermediate feature sequence. In other words, based on the second intermediate feature sequence, the sub-feature sequence with the same resolution as the first intermediate feature sequence in the second intermediate feature sequence is added to the first intermediate feature sequence to obtain the target sequence in this embodiment.

[0126] Taking into account that resolution alignment can ensure that different feature sequences have the same dimensions when performing mathematical operations or model training, this embodiment determines the self-attention matrix of each sub-feature sequence in the target sequence and uses bilinear interpolation to interpolate it so that the resolution of the target sequence is aligned with the resolution of the first intermediate feature sequence to obtain the third intermediate feature sequence. It can also avoid the additional computational burden caused by resolution misalignment and improve the operating efficiency of the algorithm.

[0127] Exemplarily, the target sequence can be calculated by the following formula: W :

[0128] , formula 3

[0129] in, represents multi-scale attention, F 1 indicates the first intermediate feature sequence.

[0130] In a possible implementation, the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence, including:

[0131] Performing attention fusion processing on the first target feature sequence and the second intermediate feature sequence to obtain a fourth intermediate feature sequence;

[0132] The fourth intermediate feature sequence, the second intermediate feature sequence and the first target feature sequence are added together to obtain a second target feature sequence.

[0133] This embodiment obtains the second target feature sequence by adding three feature sequences. Specifically, the fourth intermediate feature sequence obtained by attention fusion processing in this embodiment, and the second intermediate feature sequence and the first target feature sequence determined in the above embodiment are added to obtain the second target feature sequence, thereby achieving accurate feature extraction and fusion, and can effectively improve the accuracy and reliability of determining the target three-dimensional point cloud data.

[0134] In some embodiments, the second target feature sequence obtained after addition can be normalized to a preset dimension through a fully connected neural network to obtain target three-dimensional point cloud data. The target three-dimensional point cloud data is displayed through common visualization software, and can also assist security personnel next to the security inspection machine to identify items displayed through three-dimensional reconstruction.

[0135] Corresponding to the above method embodiment, the present application embodiment also provides a three-dimensional reconstruction system, see Figure 2 , Figure 2 A functional module diagram of a 3D reconstruction system provided in an embodiment of the present application, wherein the 3D reconstruction system 200 includes:

[0136] A first determination module 210 is used to determine an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship;

[0137] A second determining module 220 is used to determine a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed;

[0138] A first processing module 230 is used to perform attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence;

[0139] The second processing module 240 is used to input the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder;

[0140] The third processing module 250 is used to complete the three-dimensional reconstruction according to the target three-dimensional point cloud data when the target three-dimensional point cloud data is determined according to the second target feature sequence, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence.

[0141] The three-dimensional reconstruction system provided in the embodiment of the present application can achieve the following Figure 1 The various processes implemented in the method embodiments can achieve similar or identical technical effects, and will not be described again here to avoid repetition.

[0142] In a possible implementation manner, the second determining module 220 is further specifically configured to:

[0143] Divide the image to be processed to obtain multiple sub-images to be processed;

[0144] Performing linear mapping processing on the multiple sub-images to be processed to determine an initial feature sequence, wherein the initial feature sequence includes multiple one-dimensional feature sequences obtained after performing linear mapping processing on the multiple sub-images to be processed;

[0145] Position encoding of the initial feature sequence;

[0146] A first intermediate feature sequence is determined according to the initial feature sequences before and after the position encoding.

[0147] In a possible implementation, the second determining module 220 includes a first determining submodule, where the first determining submodule is configured to:

[0148] The initial feature sequences before and after position encoding are added together to obtain a first intermediate feature sequence.

[0149] In a possible implementation manner, the second determining module 220 is further specifically configured to:

[0150] Perform one-dimensional convolution of different dimensions on the image to be processed to obtain feature pyramids of different resolutions;

[0151] Input the feature pyramid to obtain feature sequences of different resolutions corresponding to the output of the preset convolutional neural network;

[0152] The feature sequences with different resolutions are serially fused to obtain a second intermediate feature sequence.

[0153] In a possible implementation manner, the second intermediate feature sequence includes a plurality of sub-feature sequences, and the first processing module 230 is further specifically configured to:

[0154] Determining the resolution of the first intermediate feature sequence and the resolution of each sub-feature sequence in the second intermediate feature sequence;

[0155] A third intermediate feature sequence is determined according to sub-feature sequences in the second intermediate feature sequence having the same resolution as that of the first intermediate feature sequence.

[0156] In a possible implementation, the first processing module 230 further includes a first processing submodule, where the first processing submodule is configured to:

[0157] Adding the first sub-feature sequence to the first intermediate feature sequence to obtain a second sub-feature sequence;

[0158] Determine a target sequence according to the second sub-feature sequence and the third sub-feature sequence, wherein the first sub-feature sequence includes a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence, and the third sub-feature sequence is a sub-feature sequence in the second intermediate feature sequence excluding the first sub-feature sequence;

[0159] According to the preset encoder, determine the self-attention matrix of each sub-feature sequence in the target sequence;

[0160] When the resolution of the target sequence is determined, all self-attention matrices are interpolated according to the bilinear interpolation method to align the resolution of the target sequence with the resolution of the first intermediate feature sequence to obtain the third intermediate feature sequence.

[0161] In a possible implementation manner, the third processing module 250 is further configured to:

[0162] Performing attention fusion processing on the first target feature sequence and the second intermediate feature sequence to obtain a fourth intermediate feature sequence;

[0163] The fourth intermediate feature sequence, the second intermediate feature sequence and the first target feature sequence are added together to obtain a second target feature sequence.

[0164] This application also provides a security inspection machine, see Figure 3 , Figure 3 The following is a diagram of the internal structure of a security inspection machine provided in an embodiment of the present application. The security inspection machine includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the security inspection machine stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the three-dimensional reconstruction method applied to the security inspection machine in the above embodiment. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may execute the three-dimensional reconstruction method. Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the security inspection machine to which the scheme of the present application is applied. The specific security inspection machine may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] The embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional reconstruction method in the method embodiment is implemented.

[0166] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the embodiment of the above-mentioned three-dimensional reconstruction method, and can achieve similar or identical technical effects. To avoid repetition, it will not be repeated here.

[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Determining an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship; Determine a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed; Performing attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence; Inputting the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder; In a case where target three-dimensional point cloud data is determined according to a second target feature sequence, completing three-dimensional reconstruction according to the target three-dimensional point cloud data, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence; Wherein, determining the first intermediate feature sequence of the image to be processed includes: Dividing the image to be processed to obtain a plurality of sub-images to be processed; Performing linear mapping processing on the multiple sub-images to be processed to determine an initial feature sequence, wherein the initial feature sequence includes multiple one-dimensional feature sequences obtained after performing linear mapping processing on the multiple sub-images to be processed; Performing position encoding on the initial feature sequence; Adding the initial feature sequences before and after the position encoding to obtain the first intermediate feature sequence; Wherein, determining the second intermediate feature sequence of the image to be processed includes: Performing one-dimensional convolution of different dimensions on the image to be processed to obtain feature pyramids of different resolutions; Inputting the feature pyramids of different resolutions into a preset convolutional neural network for processing to obtain feature sequences of different resolutions corresponding to the output of the preset convolutional neural network; The feature sequences with different resolutions are serially fused to obtain the second intermediate feature sequence.

2. The method according to claim 1, characterized in that: The second intermediate feature sequence includes a plurality of sub-feature sequences, and the attention fusion processing is performed on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence, including: Determining the resolution of the first intermediate feature sequence and the resolution of each sub-feature sequence in the second intermediate feature sequence; The third intermediate feature sequence is determined according to a sub-feature sequence in the second intermediate feature sequence having the same resolution as that of the first intermediate feature sequence.

3. The method according to claim 2, characterized in that The determining the third intermediate feature sequence according to a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence comprises: Adding the first sub-feature sequence to the first intermediate feature sequence to obtain a second sub-feature sequence; Determine a target sequence according to the second sub-feature sequence and the third sub-feature sequence, wherein the first sub-feature sequence includes a sub-feature sequence in the second intermediate feature sequence having the same resolution as the first intermediate feature sequence, and the third sub-feature sequence is a sub-feature sequence in the second intermediate feature sequence excluding the first sub-feature sequence; Determining a self-attention matrix of each sub-feature sequence in the target sequence according to the preset encoder; When the resolution of the target sequence is determined, all the self-attention matrices are interpolated according to the bilinear interpolation method to align the resolution of the target sequence with the resolution of the first intermediate feature sequence to obtain the third intermediate feature sequence.

4. The method according to claim 1, characterized in that: The second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence, including: Performing attention fusion processing on the first target feature sequence and the second intermediate feature sequence to obtain a fourth intermediate feature sequence; The fourth intermediate feature sequence, the second intermediate feature sequence and the first target feature sequence are added together to obtain the second target feature sequence.

5. A three-dimensional reconstruction system, characterized in that: The system comprises: A first determining module, configured to determine an image to be processed according to a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are arranged in a geometric relationship; A second determining module is used to determine a first intermediate feature sequence and a second intermediate feature sequence of the image to be processed, wherein the first intermediate feature sequence includes spatial information of the image to be processed, and the second intermediate feature sequence includes multi-scale feature information of the image to be processed; A first processing module, configured to perform attention fusion processing on the first intermediate feature sequence and the second intermediate feature sequence to obtain a third intermediate feature sequence; A second processing module, configured to input the first intermediate feature sequence and the third intermediate feature sequence into a preset encoder to obtain a first target feature sequence output by the preset encoder; A third processing module, configured to complete three-dimensional reconstruction according to the target three-dimensional point cloud data when the target three-dimensional point cloud data is determined according to a second target feature sequence, wherein the second target feature sequence is determined according to the first target feature sequence and the second intermediate feature sequence; Wherein, the second determining module is further used for: Dividing the image to be processed to obtain a plurality of sub-images to be processed; Performing linear mapping processing on the multiple sub-images to be processed to determine an initial feature sequence, wherein the initial feature sequence includes multiple one-dimensional feature sequences obtained after performing linear mapping processing on the multiple sub-images to be processed; Performing position encoding on the initial feature sequence; Adding the initial feature sequences before and after the position encoding to obtain the first intermediate feature sequence; The second determining module is further used for: Performing one-dimensional convolution of different dimensions on the image to be processed to obtain feature pyramids of different resolutions; Inputting the feature pyramids of different resolutions into a preset convolutional neural network for processing to obtain feature sequences of different resolutions corresponding to the output of the preset convolutional neural network; The feature sequences with different resolutions are serially fused to obtain the second intermediate feature sequence.

6. A security inspection machine, characterized in that: The security inspection machine includes a memory and a processor, the memory stores a computer program, and the computer program implements any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 4 is implemented.

Images