Target object detection method, model training method, device and electronic equipment

Abstract

The embodiments of the present application provide a target object detection method, a model training method, a device and an electronic device, and relate to the field of holographic data processing technology. The method includes: obtaining three-dimensional holographic data obtained by scanning a space to be scanned by a radar as holographic data to be detected; dividing each slice of the holographic data to be detected into multiple space blocks to be detected according to the same division method; selecting multiple space blocks to be detected that have the same position in the corresponding slice from the space blocks to be detected obtained by the division, and obtaining a space block group to be detected; encoding each space block group to be detected based on a deep self-attention mechanism; encoding the splicing result of each depth characteristic vector to be detected based on a spatial self-attention mechanism; using the obtained space feature vector to be detected to perform detection, and obtain a detection result representing the target object in the space to be scanned. In this way, automatic detection can be achieved and the efficiency of detection can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of holographic data processing, and in particular to a target object detection method, a model training method, a device and an electronic device. Background Art

[0002] Millimeter waves are electromagnetic waves with a wavelength of 1-10mm, which can penetrate most fabrics and image metals and non-metals under human clothing. Therefore, in the context of maintaining public safety, using millimeter-wave radar to detect whether people carry target objects (such as lighters and knives) in their clothing is one of the important security inspection methods. Correspondingly, millimeter-wave radar can use the above-mentioned characteristics of millimeter waves to actively transmit millimeter waves, and generate millimeter-wave images based on the received echo signals.

[0003] However, manually resolving millimeter wave images to determine target objects in the scanned area is not efficient and effective. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a target object detection method, a model training method, a device and an electronic device to improve the detection efficiency and effect of the target object. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a target object detection method is first provided, the method comprising:

[0006] Acquire three-dimensional holographic data obtained by scanning the space to be scanned by radar as the holographic data to be detected; wherein one three-dimensional holographic data includes multiple slices of the same size; one pixel point in one slice corresponds to a real position in the space to be scanned, the value of the pixel point includes characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to the pixels in one slice have the same scanning depth;

[0007] According to the same division method, each slice of the to-be-detected holographic data is divided into a plurality of to-be-detected spatial blocks;

[0008] Selecting multiple to-be-detected space blocks having the same position in the corresponding slices from the to-be-detected space blocks obtained by the division to obtain a to-be-detected space block group;

[0009] Based on the deep self-attention mechanism, each space block group to be detected is encoded respectively to obtain the depth feature vector to be detected corresponding to each space block group to be detected;

[0010] Based on the spatial self-attention mechanism, the concatenation results of the depth feature vectors to be detected are encoded to obtain the spatial feature vector to be detected;

[0011] Detection is performed using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

[0012] In some embodiments, the acquiring of three-dimensional holographic data obtained by scanning the space to be scanned by a radar as the holographic data to be detected includes:

[0013] For each preset scanning angle, the echo information obtained by the radar when scanning the space to be scanned at the preset scanning angle is obtained as the first echo information to be detected; wherein the first echo information to be detected at a preset scanning angle includes: the echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: multiple preset front scanning angles and multiple preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the space to be scanned; the preset back scanning angle represents the scanning angle when scanning the other side of the space to be scanned;

[0014] The first echo information to be detected obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second echo information to be detected at each preset front scanning angle; and the first echo information to be detected obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second echo information to be detected at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0015] Performing coherent synthesis on the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and performing coherent synthesis on the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0016] Based on the preset observation time and time anti-convolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain all-round echo information to be detected; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0017] The omnidirectional echo information to be detected is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as the holographic data to be detected.

[0018] In some embodiments, the first echo information to be detected obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second echo information to be detected at each preset front scanning angle; and the first echo information to be detected obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second echo information to be detected at each preset back scanning angle, including:

[0019] For the first echo information to be detected obtained at each preset scanning angle, the first echo information to be detected is adjusted according to the first formula to obtain the second echo information to be detected at each preset scanning angle;

[0020] Among them, the first formula is:

[0021] ;

[0022] θ represents a preset scanning angle, Indicates the observation unit in the first echo information to be detected under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second echo information to be detected under the preset scanning angle; Represents a collection; represents a natural constant;

[0023] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0024] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0025] The coherent synthesis of the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and the coherent synthesis of the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected, include:

[0026] Based on the second formula, coherently synthesize the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and coherently synthesize the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0027] Wherein, the second formula is:

[0028] ;

[0029] ;

[0030] Where θ represents a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second echo information to be detected under the preset scanning angle; Indicates the positive echo information to be detected. Indicates the back echo information to be detected; Indicates sum.

[0031] In some embodiments, the front echo information to be detected and the back echo information to be detected are fused based on the preset observation time and time deconvolution processing to obtain the omnidirectional echo information to be detected, including:

[0032] Using the third formula, based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain the omnidirectional echo information to be detected;

[0033] Wherein, the third formula is:

[0034] ;

[0035] Indicates one of the front echo information to be detected and the back echo information to be detected; represents the result obtained by processing the other of the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution; T represents the preset observation time; t represents time; Indicates the echo information to be detected in all directions.

[0036] In some embodiments, the deep self-attention mechanism is used to encode each to-be-detected spatial block group to obtain a to-be-detected depth feature vector corresponding to each to-be-detected spatial block group, including:

[0037] Input each to-be-detected spatial block group into the deep self-attention encoder in the pre-trained target object detection model to obtain the to-be-detected deep feature vector corresponding to each to-be-detected spatial block group; wherein the pre-trained target object detection model is: trained based on the sample holographic data and the label corresponding to each sample holographic data; the pre-trained target object detection model also includes: a spatial self-attention encoder;

[0038] The splicing result of each depth feature vector to be detected is encoded based on the spatial self-attention mechanism to obtain the spatial feature vector to be detected, including:

[0039] According to the position of the to-be-detected spatial slices contained in the corresponding to-be-detected spatial slice group in the corresponding slice, each to-be-detected depth feature vector is spliced ​​to obtain a to-be-detected splicing result;

[0040] The splicing result to be detected is input into the spatial self-attention encoder to obtain a spatial feature vector to be detected.

[0041] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0042] The splicing result to be detected is input into the spatial self-attention encoder to obtain the spatial feature vector to be detected, including:

[0043] Inputting the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder;

[0044] The output results of each spatial self-attention layer are fused to obtain the spatial feature vector to be detected.

[0045] In some embodiments, the pre-trained target object detection model further includes: a detection head;

[0046] The detecting by using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned includes:

[0047] Inputting the spatial feature vector to be detected into the detection head to obtain a prediction frame and a confidence level corresponding to each prediction frame; wherein the confidence level corresponding to a prediction frame indicates the probability that the prediction frame contains the target object;

[0048] If there is a prediction box whose confidence is greater than a preset confidence threshold, it is determined that the target object exists in the space to be scanned.

[0049] A second aspect of the embodiments of the present application provides a model training method, the method comprising:

[0050] Obtain the sample holographic data of the current batch and the labels of each sample holographic data; wherein the label of a sample holographic data represents the true value frame in the sample holographic data, and each true value frame represents the area occupied by a target object; a sample holographic data comprises multiple slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, the value of the pixel point comprises the characteristic information of the echo signal obtained by scanning the real position, and the real positions corresponding to each pixel point in a slice have the same scanning depth;

[0051] For each sample holographic data of the current batch, each slice in the sample holographic data is divided into a plurality of sample space blocks according to the same division method;

[0052] Selecting multiple sample space blocks having the same position in the corresponding slice from the sample space blocks obtained by the division to obtain a sample space block group;

[0053] Input each sample space block group into the deep self-attention encoder in the target object detection model in the initial state to obtain the sample depth feature vector corresponding to each sample space block group; wherein the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head;

[0054] According to the position of the sample space blocks included in the corresponding sample space block group in the corresponding slice, the depth feature vectors of each sample are spliced ​​to obtain a sample splicing result;

[0055] Input the sample splicing result into the spatial self-attention encoder to obtain a sample space feature vector;

[0056] Inputting the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data;

[0057] Calculate the loss value based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data;

[0058] Based on the obtained loss value, the model parameters of the target object detection model in the initial state are adjusted.

[0059] In some embodiments, the current batch of sample holographic data is selected from the full amount of sample holographic data; the target object detection model in the initial state is obtained by pre-training based on high-quality sample holographic data and labels corresponding to each high-quality sample holographic data; the image blurriness of the high-quality sample holographic data is lower than the image blurriness of other sample holographic data in the full amount of sample holographic data.

[0060] In some embodiments, the prediction result of a sample holographic data includes: a prediction box in the sample holographic data, and a confidence level corresponding to each prediction box; the confidence level corresponding to a prediction box indicates the probability that the prediction box contains the target object;

[0061] The calculation of the loss value based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data includes:

[0062] For each truth frame in the label of the sample holographic data, the fuzziness of the truth frame is calculated; wherein the fuzziness of a truth frame represents: the detection difficulty of the current target object detection model for the target object in the truth frame;

[0063] The loss value is calculated using the position difference between a true value box whose ambiguity is not greater than the current ambiguity threshold and a prediction box corresponding to the true value box, as well as the confidence corresponding to the prediction box.

[0064] In some embodiments, the method further comprises:

[0065] After adjusting the model parameters of the target object detection model in the initial state based on the loss value obtained from the current batch of sample holographic data, the current blur threshold is increased.

[0066] In some embodiments, the step of calculating the fuzziness of each truth frame in the label of the sample holographic data comprises:

[0067] For each true value frame in the label of the sample holographic data, respectively calculate the intersection-over-union ratio between the true value frame and each predicted frame;

[0068] Calculate the product of each intersection-over-union ratio and the confidence of the prediction box used to obtain the intersection-over-union ratio;

[0069] The inverse of the maximum value in each product is determined as the fuzziness of the true value frame.

[0070] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0071] The sample concatenation result is input into the spatial self-attention encoder to obtain a sample space feature vector, including:

[0072] Inputting the sample splicing result into the first spatial self-attention layer in the spatial self-attention encoder;

[0073] The output results of each spatial self-attention layer are fused to obtain the sample space feature vector.

[0074] In some embodiments, each sample holographic data is obtained by the following steps:

[0075] For each preset scanning angle, echo information obtained by the radar scanning the sample space at the preset scanning angle is obtained as first sample echo information; wherein the first sample echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the sample space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the sample space;

[0076] The first sample echo information obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0077] Performing coherent synthesis on the second sample echo information at each preset front scanning angle to obtain front sample echo information, and performing coherent synthesis on the second sample echo information at each preset back scanning angle to obtain back sample echo information;

[0078] Based on the preset observation time and time anti-convolution processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0079] The omnidirectional sample echo information is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as sample holographic data.

[0080] In some embodiments, the first sample echo information obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle, including:

[0081] For the first sample echo information obtained at each preset scanning angle, the first sample echo information is adjusted according to the fourth formula to obtain the second sample echo information at each preset scanning angle;

[0082] Wherein, the fourth formula is:

[0083] ;

[0084] Indicates a preset scanning angle, Indicates the observation unit in the first sample echo information under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second sample echo information under the preset scanning angle; Represents a collection; represents a natural constant;

[0085] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0086] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0087] The coherent synthesis of the second sample echo information at each preset front scanning angle to obtain the front sample echo information, and the coherent synthesis of the second sample echo information at each preset back scanning angle to obtain the back sample echo information, include:

[0088] Based on the fifth formula, the second sample echo information at each preset front scanning angle is coherently synthesized to obtain the front sample echo information, and the second sample echo information at each preset back scanning angle is coherently synthesized to obtain the back sample echo information;

[0089] Wherein, the fifth formula is:

[0090] ;

[0091] ;

[0092] in, Indicates a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second sample echo information at the preset scanning angle at the preset scanning angle; Indicates the positive sample echo information, Indicates the back sample echo information; Indicates sum.

[0093] In some embodiments, the front sample echo information and the back sample echo information are fused based on the preset observation time and time anti-convolution processing to obtain omnidirectional sample echo information, including:

[0094] Using the sixth formula, based on the preset observation time and time anti-folding processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information;

[0095] Wherein, the sixth formula is:

[0096] ;

[0097] Indicates one of the front sample echo information and the back sample echo information; represents the result obtained by processing the other sample echo information of the front sample echo information and the back sample echo information based on the preset observation time and time anti-convolution; T represents the preset observation time; t represents the time interval; Represents omni-directional sample echo information.

[0098] According to a third aspect of the present application, a target object detection device is provided, the device comprising:

[0099] The holographic data acquisition unit to be detected is used to acquire three-dimensional holographic data obtained by scanning the space to be scanned by radar as the holographic data to be detected; wherein one three-dimensional holographic data includes multiple slices of the same size; one pixel point in one slice corresponds to a real position in the space to be scanned, the value of the pixel point includes characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to the pixels in one slice have the same scanning depth;

[0100] A first division unit, used for dividing each slice of the to-be-detected holographic data into a plurality of to-be-detected spatial blocks according to the same division method;

[0101] A first grouping unit is used to select a plurality of to-be-detected spatial blocks having the same position in the corresponding slice from the to-be-detected spatial blocks obtained by division, to obtain a to-be-detected spatial block group;

[0102] A first depth encoding unit is used to encode each to-be-detected spatial block group based on a deep self-attention mechanism to obtain a to-be-detected depth feature vector corresponding to each to-be-detected spatial block group;

[0103] A first spatial encoding unit is used to encode the splicing results of the obtained depth feature vectors to be detected based on a spatial self-attention mechanism to obtain a spatial feature vector to be detected;

[0104] The detection result acquisition unit is used to perform detection using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

[0105] In some embodiments, the unit for acquiring holographic data to be detected includes:

[0106] The first to-be-detected echo information acquisition subunit is used to acquire, for each preset scanning angle, echo information obtained by the radar when scanning the to-be-scanned space at the preset scanning angle as the first to-be-detected echo information; wherein the first to-be-detected echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the to-be-scanned space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the to-be-scanned space;

[0107] The second to-be-detected echo information acquisition subunit is used to adjust the first to-be-detected echo information obtained at each preset front scanning angle to the same specified front scanning angle, and obtain the second to-be-detected echo information at each preset front scanning angle; and adjust the first to-be-detected echo information obtained at each preset back scanning angle to the same specified back scanning angle, and obtain the second to-be-detected echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0108] The first coherent synthesis subunit is used to perform coherent synthesis on the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and to perform coherent synthesis on the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0109] The first fusion subunit is used to fuse the front echo information to be detected and the back echo information to be detected based on the preset observation time and time anti-convolution processing to obtain the omnidirectional echo information to be detected; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0110] The to-be-detected holographic data acquisition subunit is used to process the omnidirectional to-be-detected echo information based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as the to-be-detected holographic data.

[0111] In some embodiments, the second to-be-detected echo information acquisition subunit is specifically used to:

[0112] For the first echo information to be detected obtained at each preset scanning angle, the first echo information to be detected is adjusted according to the first formula to obtain the second echo information to be detected at each preset scanning angle;

[0113] Among them, the first formula is:

[0114] ;

[0115] θ represents a preset scanning angle, Indicates the observation unit in the first echo information to be detected under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second echo information to be detected under the preset scanning angle; Represents a collection; represents a natural constant;

[0116] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0117] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0118] The first coherent synthesis subunit is specifically used for:

[0119] Based on the second formula, coherently synthesize the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and coherently synthesize the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0120] Wherein, the second formula is:

[0121] ;

[0122] ;

[0123] Where θ represents a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second echo information to be detected under the preset scanning angle; Indicates the positive echo information to be detected. Indicates the back echo information to be detected; Indicates sum.

[0124] In some embodiments, the first fusion subunit is specifically used for:

[0125] Using the third formula, based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain the omnidirectional echo information to be detected;

[0126] Wherein, the third formula is:

[0127] ;

[0128] Indicates one of the front echo information to be detected and the back echo information to be detected; represents the result obtained by processing the other of the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution; T represents the preset observation time; t represents time; Indicates the echo information to be detected in all directions.

[0129] In some embodiments, the first depth encoding unit is specifically configured to:

[0130] Input each to-be-detected spatial block group into the deep self-attention encoder in the pre-trained target object detection model to obtain the to-be-detected deep feature vector corresponding to each to-be-detected spatial block group; wherein the pre-trained target object detection model is: trained based on the sample holographic data and the label corresponding to each sample holographic data; the pre-trained target object detection model also includes: a spatial self-attention encoder;

[0131] The first spatial encoding unit includes:

[0132] The detection splicing result acquisition subunit is used to splice the depth feature vectors to be detected according to the positions of the to-be-detected spatial blocks contained in the corresponding to-be-detected spatial block group in the corresponding slice to obtain the to-be-detected splicing result;

[0133] The first spatial encoding subunit is used to input the splicing result to be detected into the spatial self-attention encoder to obtain a spatial feature vector to be detected.

[0134] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0135] The first spatial encoding subunit is specifically used for:

[0136] Inputting the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder;

[0137] The output results of each spatial self-attention layer are fused to obtain the spatial feature vector to be detected.

[0138] In some embodiments, the pre-trained target object detection model further includes: a detection head;

[0139] The detection result acquisition unit is specifically used for:

[0140] Inputting the spatial feature vector to be detected into the detection head to obtain a prediction frame and a confidence level corresponding to each prediction frame; wherein the confidence level corresponding to a prediction frame indicates the probability that the prediction frame contains the target object;

[0141] If there is a prediction box whose confidence is greater than a preset confidence threshold, it is determined that the target object exists in the space to be scanned.

[0142] In a fourth aspect of the embodiments of the present application, a model training device is provided, the device comprising:

[0143] The sample holographic data acquisition unit is used to acquire the sample holographic data of the current batch and the labels of each sample holographic data; wherein the label of a sample holographic data represents a truth frame in the sample holographic data, and each truth frame represents an area occupied by a target object; a sample holographic data comprises a plurality of slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, the value of the pixel point comprises characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to each pixel point in a slice have the same scanning depth;

[0144] A second division unit is used for dividing each slice of each sample holographic data of the current batch into a plurality of sample space blocks according to the same division method;

[0145] A second grouping unit is used to select a plurality of sample space blocks having the same position in the corresponding slice from the sample space blocks obtained by the division, so as to obtain a sample space block group;

[0146] A second depth encoding unit is used to input each sample space block group into a deep self-attention encoder in the target object detection model in the initial state to obtain a sample depth feature vector corresponding to each sample space block group; wherein the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head;

[0147] A sample splicing result acquisition unit is used to splice the depth feature vectors of each sample according to the position of the sample space blocks included in the corresponding sample space block group in the corresponding slice to obtain a sample splicing result;

[0148] A second spatial encoding unit, used for inputting the sample splicing result into the spatial self-attention encoder to obtain a sample spatial feature vector;

[0149] A prediction result acquisition unit, used for inputting the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data;

[0150] a loss value calculation unit, configured to calculate a loss value based on a difference between a prediction result of the sample holographic data and a label corresponding to the sample holographic data;

[0151] The model parameter adjustment unit is used to adjust the model parameters of the target object detection model in the initial state based on the obtained loss value.

[0152] In some embodiments, the current batch of sample holographic data is selected from the full amount of sample holographic data; the target object detection model in the initial state is obtained by pre-training based on high-quality sample holographic data and labels corresponding to each high-quality sample holographic data; the image blurriness of the high-quality sample holographic data is lower than the image blurriness of other sample holographic data in the full amount of sample holographic data.

[0153] In some embodiments, the prediction result of a sample holographic data includes: a prediction box in the sample holographic data, and a confidence level corresponding to each prediction box; the confidence level corresponding to a prediction box indicates the probability that the prediction box contains the target object;

[0154] The loss value calculation unit includes:

[0155] The fuzziness calculation subunit is used to calculate the fuzziness of each truth frame in the label of the sample holographic data; wherein the fuzziness of a truth frame represents: the detection difficulty of the current target object detection model for the target object in the truth frame;

[0156] The loss value calculation subunit is used to calculate the loss value by using the position difference between a true value frame whose ambiguity is not greater than the current ambiguity threshold and a prediction frame corresponding to the true value frame, as well as the confidence corresponding to the prediction frame.

[0157] In some embodiments, the apparatus further comprises:

[0158] The fuzziness threshold adjustment unit is used to increase the current fuzziness threshold after adjusting the model parameters of the target object detection model in the initial state based on the loss value obtained from the sample holographic data of the current batch.

[0159] In some embodiments, the ambiguity calculation subunit is specifically used to:

[0160] For each true value frame in the label of the sample holographic data, respectively calculate the intersection-over-union ratio between the true value frame and each predicted frame;

[0161] Calculate the product of each intersection-over-union ratio and the confidence of the prediction box used to obtain the intersection-over-union ratio;

[0162] The inverse of the maximum value in each product is determined as the fuzziness of the true value frame.

[0163] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0164] The second spatial coding unit is specifically configured to:

[0165] Inputting the sample splicing result into the first spatial self-attention layer in the spatial self-attention encoder;

[0166] The output results of each spatial self-attention layer are fused to obtain the sample space feature vector.

[0167] In some embodiments, each sample holographic data is obtained by the following steps:

[0168] For each preset scanning angle, echo information obtained by the radar scanning the sample space at the preset scanning angle is obtained as first sample echo information; wherein the first sample echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the sample space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the sample space;

[0169] The first sample echo information obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0170] Performing coherent synthesis on the second sample echo information at each preset front scanning angle to obtain front sample echo information, and performing coherent synthesis on the second sample echo information at each preset back scanning angle to obtain back sample echo information;

[0171] Based on the preset observation time and time anti-convolution processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0172] The omnidirectional sample echo information is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as sample holographic data.

[0173] In some embodiments, the first sample echo information obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle, including:

[0174] For the first sample echo information obtained at each preset scanning angle, the first sample echo information is adjusted according to the fourth formula to obtain the second sample echo information at each preset scanning angle;

[0175] Wherein, the fourth formula is:

[0176] ;

[0177] Indicates a preset scanning angle, Indicates the observation unit in the first sample echo information under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second sample echo information under the preset scanning angle; Represents a collection; represents a natural constant;

[0178] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0179] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0180] The coherent synthesis of the second sample echo information at each preset front scanning angle to obtain the front sample echo information, and the coherent synthesis of the second sample echo information at each preset back scanning angle to obtain the back sample echo information, include:

[0181] Based on the fifth formula, the second sample echo information at each preset front scanning angle is coherently synthesized to obtain the front sample echo information, and the second sample echo information at each preset back scanning angle is coherently synthesized to obtain the back sample echo information;

[0182] Wherein, the fifth formula is:

[0183] ;

[0184] ;

[0185] in, Indicates a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second sample echo information at the preset scanning angle at the preset scanning angle; Indicates the positive sample echo information, Indicates the back sample echo information; Indicates sum.

[0186] In some embodiments, the front sample echo information and the back sample echo information are fused based on the preset observation time and time anti-convolution processing to obtain omnidirectional sample echo information, including:

[0187] Using the sixth formula, based on the preset observation time and time anti-folding processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information;

[0188] Wherein, the sixth formula is:

[0189] ;

[0190] Indicates one of the front sample echo information and the back sample echo information; represents the result obtained by processing the other sample echo information of the front sample echo information and the back sample echo information based on the preset observation time and time anti-convolution; T represents the preset observation time; t represents the time interval; Represents omni-directional sample echo information.

[0191] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including:

[0192] Memory, used to store computer programs;

[0193] The processor is used to implement any of the target object detection methods or model training methods described above when executing the program stored in the memory.

[0194] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the target object detection methods or model training methods described above is implemented.

[0195] In another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the target object detection methods or model training methods described above.

[0196] Beneficial effects of the embodiments of the present application:

[0197] Based on the target object detection method provided in the embodiment of the present application, after scanning the space to be scanned by radar (i.e., millimeter wave radar) and obtaining three-dimensional holographic data (i.e., holographic data to be detected), the electronic device can automatically process the holographic data to be detected, and perform detection based on the feature vector of the space to be detected obtained by processing, thereby realizing automated detection and improving detection efficiency.

[0198] Furthermore, during the detection process, the electronic device can divide each slice in the holographic data to be detected in the same way, and divide each slice into multiple spatial blocks to be detected. Since the scanning depth of the real position corresponding to each pixel in a slice is the same, that is, each pixel in the slice belongs to the same scanning plane, that is, the coordinate value of each pixel in the slice in the depth direction is the same. The real position corresponding to a pixel can be represented by a coordinate value in the depth direction and a coordinate value of the pixel in the scanning plane (that is, a horizontal coordinate value and a vertical coordinate value), and the coordinate value combination corresponding to any two pixels is different. Therefore, the multiple spatial blocks to be detected obtained by dividing a slice correspond to different areas in a scanning plane, and the different areas in a scanning plane respectively include: pixels in different horizontal coordinate value ranges and vertical coordinate value ranges at the scanning depth. Then, multiple spatial blocks to be detected that are consistent in position in the slice to which they belong are selected from the spatial blocks to be detected obtained by the division to obtain a group of spatial blocks to be detected. Since the size of each slice is consistent and the division method is the same, each space block to be detected in a space block group to be detected contains: the difference between the values ​​of each pixel point in the same area (i.e., the horizontal coordinate value range is consistent and the vertical coordinate value range is consistent) in the scanning plane at different scanning depths. Accordingly, each space block group to be detected is encoded based on the deep self-attention mechanism, and the encoding result is processed based on the spatial self-attention mechanism, so that the characteristics of the holographic data to be detected in the depth dimension and the spatial dimension can be learned, and the encoding result of the spatial self-attention mechanism is used for detection to obtain the detection result representing the target object in the space to be scanned, so that the accuracy of detection can be improved.

[0199] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0200] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0201] Figure 1 A first flow chart of a target object detection method provided in an embodiment of the present application;

[0202] FIG2 (a) is a schematic diagram of a planar scanning radar;

[0203] FIG2( b ) is an example diagram of a visualization result obtained by visualizing scanning data of a planar scanning radar;

[0204] FIG3 (a) is a schematic diagram of a cylindrical scanning radar;

[0205] FIG3( b ) is an example diagram of a visualization result obtained by visualizing the scanning data of a cylindrical scanning radar;

[0206] Figure 4 A schematic diagram of three-dimensional holographic data provided in an embodiment of the present application;

[0207] Figure 5 An example diagram of dividing a slice provided in an embodiment of the present application;

[0208] Figure 6 A second flow chart of the target object detection method provided in an embodiment of the present application;

[0209] Figure 7 A schematic diagram of a depth-dimensional data slice provided in an embodiment of the present application;

[0210] Figure 8 A schematic diagram of the structure of a spatial self-attention encoder provided in an embodiment of the present application;

[0211] Fig. 9 A third flow chart of the target object detection method provided in an embodiment of the present application;

[0212] Fig.10 A schematic diagram of a process for detecting a target object based on three-dimensional holographic data provided in an embodiment of the present application;

[0213] Fig.11 A flow chart for obtaining holographic data to be detected provided in an embodiment of the present application;

[0214] Fig.12 A schematic diagram of a scene scanned by a radar at various preset scanning angles provided in an embodiment of the present application from a top view;

[0215] Fig.13 A schematic diagram of a process for obtaining holographic data to be detected provided in an embodiment of the present application;

[0216] Fig.14 A first flow chart of the model training method provided in the embodiment of the present application;

[0217] Fig.15 An example diagram of a visualization result obtained by performing visualization processing on three-dimensional holographic data provided in an embodiment of the present application;

[0218] Fig.16 An example flow chart of a target object detection model training process provided in this application;

[0219] Fig.17 An example diagram of the overall process of the target object detection model training process and prediction process provided in this application;

[0220] Fig.18 A structural diagram of a target object detection device provided in an embodiment of the present application;

[0221] Fig.19 A structural diagram of a model training device provided in an embodiment of the present application;

[0222] Fig. 20 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0223] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0224] Millimeter waves are electromagnetic waves with a wavelength of 1-10mm, which can penetrate most fabrics and image metal and non-metal under human clothing. Compared with the traditional manual security inspection method, millimeter wave radar (i.e., radar in this application) has the characteristics of non-contact perception, and compared with the security inspection method using detection sources such as X-rays, millimeter wave radar has no radiation hazard to the human body.

[0225] Therefore, in the context of maintaining public safety, using millimeter wave radar (also called millimeter wave scanning equipment) to detect whether a person is carrying a target object (such as a lighter or a knife) in their clothing is one of the important security inspection methods. Accordingly, the millimeter wave radar can use the above-mentioned characteristics of millimeter waves to actively transmit millimeter waves, and generate millimeter wave images based on the received echo signals.

[0226] However, manually resolving millimeter wave images to determine target objects in the scanning area is not efficient and effective.

[0227] The embodiment of the present application provides a method for detecting a target object, which can be applied to an electronic device, wherein the electronic device can communicate with a radar to obtain a scanning result (ie, an echo signal) of the radar.

[0228] See also Figure 1 , Figure 1 The first flow chart of the target object detection method provided in the embodiment of the present application includes the following steps:

[0229] S101: Acquire three-dimensional holographic data obtained by scanning a space to be scanned by a radar as holographic data to be detected.

[0230] Among them, a three-dimensional holographic data contains multiple slices of consistent size; a pixel point in a slice corresponds to a real position in the space to be scanned, the value of the pixel point contains characteristic information of the echo signal obtained by scanning the real position, and the scanning depth of the real position corresponding to each pixel point in a slice is the same.

[0231] S102: Divide each slice of the to-be-detected holographic data into a plurality of to-be-detected spatial blocks in the same division manner.

[0232] S103: Select multiple to-be-detected spatial blocks whose positions are consistent in the corresponding slices from the to-be-detected spatial blocks obtained by division to obtain a to-be-detected spatial block group.

[0233] S104: Encode each to-be-detected spatial block group based on the deep self-attention mechanism to obtain a to-be-detected deep feature vector corresponding to each to-be-detected spatial block group.

[0234] S105: Encode the concatenation results of the depth feature vectors to be detected based on the spatial self-attention mechanism to obtain the spatial feature vector to be detected.

[0235] S106: Perform detection using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

[0236] Based on the above processing, after scanning the space to be scanned by radar (i.e., millimeter wave radar) and obtaining three-dimensional holographic data (i.e., holographic data to be detected), the electronic device can automatically process the holographic data to be detected, and perform detection based on the feature vector of the space to be detected obtained by the processing, thereby realizing automated detection and improving detection efficiency.

[0237] Furthermore, during the detection process, the electronic device can divide each slice in the holographic data to be detected in the same way, and divide each slice into multiple spatial blocks to be detected. Since the scanning depth of the real position corresponding to each pixel in a slice is the same, that is, each pixel in the slice belongs to the same scanning plane, that is, the coordinate value of each pixel in the slice in the depth direction is the same. The real position corresponding to a pixel can be represented by a coordinate value in the depth direction and a coordinate value of the pixel in the scanning plane (that is, a horizontal coordinate value and a vertical coordinate value), and the coordinate value combination corresponding to any two pixels is different. Therefore, the multiple spatial blocks to be detected obtained by dividing a slice correspond to different areas in a scanning plane, and the different areas in a scanning plane respectively include: pixels in different horizontal coordinate value ranges and vertical coordinate value ranges at the scanning depth. Then, multiple spatial blocks to be detected that are consistent in position in the slice to which they belong are selected from the spatial blocks to be detected obtained by the division to obtain a group of spatial blocks to be detected. Since the size of each slice is consistent and the division method is the same, each space block to be detected in a space block group to be detected contains: the difference between the values ​​of each pixel point in the same area (i.e., the horizontal coordinate value range is consistent and the vertical coordinate value range is consistent) in the scanning plane at different scanning depths. Accordingly, each space block group to be detected is encoded based on the deep self-attention mechanism, and the encoding result is processed based on the spatial self-attention mechanism, so that the characteristics of the holographic data to be detected in the depth dimension and the spatial dimension can be learned, and the encoding result of the spatial self-attention mechanism is used for detection to obtain the detection result representing the target object in the space to be scanned, so that the accuracy of detection can be improved.

[0238] The radar in this application is a millimeter wave radar. According to different scanning formats, radars can be divided into plane scanning and cylindrical scanning. See Figure 2 (a), which is a physical schematic diagram of a plane scanning radar. In the above-mentioned security inspection scenario, the person who needs to undergo security inspection can stand in the space to be scanned by the plane scanning radar. Then, the plane scanning radar can scan the front and back of the human body in the space to be scanned respectively, and obtain the scanning data corresponding to the front of the human body, and the scanning data corresponding to the back of the human body (i.e., the echo signal). Subsequently, the electronic device can visualize each scanning data to obtain the visualization result of each scanning data. See Figure 2 (b), which is an example diagram of the visualization result obtained by visualizing the scanning data of the plane scanning radar. The left image in Figure 2 (b) is the visualization result of the scanning data corresponding to the front of the human body; the right image in Figure 2 (b) is the visualization result of the scanning data corresponding to the back of the human body.

[0239] See Figure 3 (a), which is a physical schematic diagram of a cylindrical scanning radar. In the above security inspection scenario, the person who needs to undergo security inspection can stand in the space to be scanned by the cylindrical scanning radar. The cylindrical scanning radar can use a scanning array that rotates with the cylindrical wall (the arrow in the figure indicates the direction of rotation) to scan the space to be scanned and obtain scanning data (i.e., echo signals) at multiple angles. Subsequently, the electronic device can visualize each scanning data to obtain a visualization result of each scanning data. See Figure 3 (b), which is an example diagram of a visualization result obtained by visualizing the scanning data of a cylindrical scanning radar. Figure 3 (b) contains a group of (7) images, which are obtained by visualizing the scanning data at different angles.

[0240] The target object detection method provided in the present application is particularly suitable for processing the scanning results of the above-mentioned cylindrical scanning radar. The processing process can be found in the subsequent embodiments.

[0241] With respect to step S101 , the space to be scanned represents: a spatial region in the real world where target object detection is required.

[0242] It is understandable that after the radar is fixedly installed at a location in the real world, the spatial area that the radar can scan is often fixed due to the limitations of the radar's installation location, installation angle, and hardware conditions. That is, the space to be scanned corresponding to the radar is fixed. Therefore, if it is necessary to obtain a scanning result for a scanning object, the radar can scan the space to be scanned when the scanning object is located in the space to be scanned, and the scanning result for the scanning object can be obtained.

[0243] In the above security inspection scenario, when the object to be scanned is a person, the person who needs to be inspected can stand in the space to be scanned, and then the radar can emit millimeter waves to the space to be scanned for scanning. Alternatively, when the object to be scanned is a package, the package can be placed in the space to be scanned, and then the radar can emit millimeter waves to the space to be scanned for scanning.

[0244] Correspondingly, the holographic data to be detected is: three-dimensional holographic data obtained based on the scanning result (i.e., echo signal) of the space to be scanned by the radar. Among them, the holographic data means: three-dimensional data that records all information such as amplitude, phase, and distance. The holographic data is obtained after the radar echo signal is holographically imaged. The specific process of obtaining the holographic data to be detected will be described in the subsequent embodiments.

[0245] A 3D holographic data contains multiple slices of the same size. The scanning depth of the real position corresponding to each pixel in a slice is the same, that is, each pixel in the slice belongs to the same scanning plane, and the coordinate values ​​of each pixel in the slice in the depth direction are the same. The real position corresponding to a pixel can be represented by a combination of a horizontal coordinate value, a vertical coordinate value, and a coordinate value in the depth direction, and the coordinate value combinations corresponding to any two pixels are different. The depth direction, horizontal direction, and vertical direction are perpendicular to each other.

[0246] See also Figure 4 , Figure 4 A schematic diagram of three-dimensional holographic data provided in an embodiment of the present application. Figure 4 In the figure, the X-axis, Y-axis, and Z-axis represent a dimension of the three-dimensional holographic data respectively. The Z-axis represents the depth dimension of the three-dimensional holographic data, and the Z-axis direction represents the above-mentioned depth direction. The X-axis represents the horizontal spatial dimension of the three-dimensional holographic data, and the X-axis direction represents the above-mentioned horizontal direction. The Y-axis represents the vertical spatial dimension of the three-dimensional holographic data, and the Y-axis direction represents the above-mentioned vertical direction. The X-axis, Y-axis, and Z-axis are perpendicular to each other, and the XOY plane (the plane formed by the X-axis and the Y-axis) is a scanning plane perpendicular to the depth direction.

[0247] The sizes of multiple slices contained in a three-dimensional holographic data are consistent, that is, the number of pixels contained in each slice is consistent. Among them, one pixel represents one observation unit. Correspondingly, the minimum and maximum values ​​of the multiple slices in the horizontal direction are the same, and the minimum and maximum values ​​of the multiple slices in the vertical direction are the same. That is, the values ​​of each slice in the horizontal spatial dimension (denoted as M) are consistent, and the values ​​of each slice in the vertical spatial dimension (denoted as N) are consistent. Accordingly, the size of a slice can be denoted as M×N. The number of slices (that is, the value of the depth dimension of the three-dimensional holographic data) is determined by the hardware conditions of the radar.

[0248] For each slice, a pixel in the slice corresponds to a real position in the space to be scanned. The number of pixels contained in a slice is determined by the size of the space to be scanned and the size of the radar's observation unit. For the same size of the space to be scanned, the smaller the radar's observation unit, the more pixels a slice contains.

[0249] The value of a pixel point in a slice contains characteristic information of an echo signal obtained by scanning the real position. For example, the characteristic information of the echo signal may include: phase and amplitude.

[0250] The scanning depth of the real position corresponding to each pixel point in a slice is the same. Figure 4 On this basis, the projection positions of each pixel point in a slice in the Z-axis direction are consistent.

[0251] With respect to step S102 to step S103, for each slice in the to-be-detected holographic data, the electronic device may perform patching processing on the slice in the same division manner, and divide the slice into a plurality of spatial patches, ie, spatial patches to be detected.

[0252] The multiple spatial blocks obtained by dividing a slice may be: multiple rectangular areas of the same size in the slice. That is, the electronic device may divide each slice into a specified number of spatial blocks equally according to the size of the slice. The size of the specified number may be set according to actual needs and is not limited here.

[0253] See also Figure 5 , Figure 5 An example diagram of dividing a slice provided in an embodiment of the present application. If the size of a slice is 90×90, for a slice, the slice can be equally divided into 9 30×30 spatial blocks, and each spatial block can be numbered according to its position in the slice. For example, the spatial blocks can be numbered in the order from top to bottom and from left to right in the slice, and the numbering is obtained as spatial block 1 to be detected, spatial block 2 to be detected, ..., spatial block 9 to be detected. Figure 5 In the figure, rectangle 1 represents the space block 1 to be detected, rectangle 2 represents the space block 2 to be detected, rectangle 3 represents the space block 3 to be detected, rectangle 4 represents the space block 4 to be detected, rectangle 5 represents the space block 5 to be detected, rectangle 6 represents the space block 6 to be detected, rectangle 7 represents the space block 7 to be detected, rectangle 8 represents the space block 8 to be detected, and rectangle 9 represents the space block 9 to be detected.

[0254] Furthermore, after each slice is divided, the electronic device can select multiple spatial blocks to be detected that have the same position in the slice to which they belong from the spatial blocks to be detected obtained by the division, and obtain a spatial block group to be detected. That is to say, for each spatial block to be detected in a spatial block group to be detected, the position of the spatial block to be detected in the slice to which it belongs is consistent with the positions of other spatial blocks to be detected in the spatial block group to be detected except for the spatial block to be detected in the respective slices to which they belong. That is, each spatial block to be detected in a spatial block group to be detected includes: the values ​​of pixel points in the same area (i.e., the horizontal coordinate value range is consistent and the vertical coordinate value range is consistent) in the scanning plane at different scanning depths.

[0255] For example, in the above Figure 5In the case of the division method and numbering method shown, if the holographic data to be detected contains: 256 90×90 slices, for each slice, the slice is equally divided into 9 30×30 spatial blocks, then a spatial block group to be detected contains: 256 spatial blocks to be detected. The 256 spatial blocks to be detected belong to slices of different scanning depths, and the numbers of the spatial blocks to be detected in the spatial block group to be detected are also the same. Correspondingly, for the above holographic data to be detected, 9 spatial block groups to be detected can be obtained.

[0256] For steps S104 to S105, after obtaining the spatial block groups to be detected corresponding to the holographic data to be detected, the electronic device can encode each spatial block group to be detected based on the deep self-attention mechanism to obtain the depth feature vector to be detected corresponding to each spatial block group to be detected.

[0257] That is, the electronic device can perform self-attention encoding on the data corresponding to the same area in the scanning plane in the vertical depth direction at different scanning depths. In this way, the correlation between each space block to be detected in a space block group to be detected in the depth dimension can be learned.

[0258] Furthermore, after obtaining the depth feature vector to be detected corresponding to each spatial block group to be detected, the depth feature vectors to be detected can be spliced ​​according to the positions of each spatial block to be detected in the spatial block group to be detected in the corresponding slice to obtain the splicing result of the depth feature vectors to be detected.

[0259] Correspondingly, the electronic device can encode the splicing results of the depth feature vectors to be detected based on the spatial self-attention mechanism to obtain the spatial feature vector to be detected.

[0260] In some embodiments, the above-mentioned depth feature vector to be detected and the space feature vector to be detected can be obtained by a pre-trained target object detection model. Figure 6 , Figure 6 A second flow chart of the target object detection method provided in the embodiment of the present application, in Figure 1 On the basis of, step S104 comprises:

[0261] S1041: Input each to-be-detected spatial block group into the deep self-attention encoder in the pre-trained target object detection model to obtain a to-be-detected deep feature vector corresponding to each to-be-detected spatial block group.

[0262] Among them, the pre-trained target object detection model is: obtained by training based on sample holographic data and labels corresponding to each sample holographic data; the pre-trained target object detection model also includes: a spatial self-attention encoder.

[0263] Step S105 includes:

[0264] S1051: According to the positions of the to-be-detected spatial blocks contained in the corresponding to-be-detected spatial block group in the corresponding slice, each to-be-detected depth feature vector is spliced ​​to obtain a to-be-detected splicing result.

[0265] S1052: Input the splicing result to be detected into the spatial self-attention encoder to obtain the spatial feature vector to be detected.

[0266] In an embodiment of the present application, the backbone network of the pre-trained target object detection model includes: a deep self-attention encoder and a spatial self-attention encoder. For example, the target object detection model can be implemented based on a Transformer network architecture.

[0267] Among them, the target object detection model is: obtained by training based on sample holographic data and the labels corresponding to each sample holographic data; the training process of the target object detection model will be explained in subsequent embodiments.

[0268] The deep self-attention encoder contains multiple deep self-attention modules, and each deep self-attention module is composed of multiple network layers. For example, in a deep self-attention module, the first layer is the Depth Self-Attension layer, the second layer is the Add&Norm layer, the third layer is the Feed Forward layer, and the fourth layer is the Add&Norm layer.

[0269] For each spatial block group to be detected, the electronic device can input the spatial block group to be detected into a deep self-attention encoder in a pre-trained target object detection model to obtain a deep feature vector to be detected corresponding to the spatial block group to be detected.

[0270] In one implementation, the electronic device can perform a linear transformation on the spatial block group to be detected to obtain input data of the deep self-attention encoder in the target object detection model. Accordingly, step S1041 includes: for each spatial block group to be detected, three different linear transformation processes are performed on the spatial block group to be detected; the results of the transformation process are respectively used as the key (K, Key), query (Q, Query), and value (V, Value), and input into the deep self-attention encoder in the pre-trained target object detection model to obtain the deep feature vector to be detected.

[0271] In the embodiment of the present application, the processing process of the deep self-attention encoder on the input data (ie, K, Q, V) can be expressed as:

[0272] ;

[0273] in, represents the output of the deep self-attention encoder, are feature vectors representing the depth direction (the direction represented by the Z axis) obtained by performing three different linear transformations on the spatial block group to be detected; It indicates that each of the to-be-detected spatial blocks in the to-be-detected spatial block group is located at the i-th position in the horizontal direction and the j-th position in the vertical direction in the slice to which it belongs; Represents the feature dimension of the key; Represents the activation function.

[0274] Correspondingly, for the input data corresponding to each spatial block group to be detected, the deep self-attention encoder can output the depth feature vector to be detected corresponding to the spatial block group to be detected. Among them, the number of depth feature vectors to be detected corresponding to a spatial block group to be detected is less than the number of spatial blocks to be detected in the spatial block group to be detected. That is, the deep self-attention encoder can compress the data corresponding to each spatial block group to be detected in the depth dimension (that is, the data corresponding to the same area in the scanning plane in the vertical depth direction at different scanning depths).

[0275] See also Figure 7 , Figure 7 A schematic diagram of a depth-dimensional data slice provided in an embodiment of the present application. Figure 7 The depth dimension data slice shown represents: for a three-dimensional holographic data, the information of the pixel points at the same horizontal and vertical spatial positions at different scanning depths, that is, the horizontal coordinate values ​​of the pixel points at each scanning depth are the same and the vertical coordinate values ​​are the same. Among them, in the depth dimension direction, the proportion of valid target information is low, and the proportion of invalid redundant information is high. Correspondingly, the deep self-attention encoder can retain the valid target information (including the maximum value point) in the data slice and filter out the invalid redundant information in the data slice. In this way, the data amount of the depth feature vector to be detected corresponding to each output spatial block group to be detected can be reduced, and the depth dimension information represented by each depth feature vector to be detected can be compressed.

[0276] It is understandable that 3D holographic data is dense 3D data with a large amount of data. If the 3D holographic data is directly processed through a feature extraction network, a large amount of computing resources will be consumed, and the model will be difficult to converge during the training process.

[0277] Based on the above processing, we can combine the characteristics of 3D holographic data (3D holographic data has high data redundancy in the depth dimension) and compress the data corresponding to each spatial block group to be detected in the depth dimension based on the deep self-attention encoder to filter out the invalid redundant information in the 3D holographic data. This will reduce the amount of calculation for model processing and improve the efficiency of model processing in the subsequent processing flow.

[0278] Moreover, in the process of encoding each spatial slice group to be detected by the deep self-attention encoder, the weight of the deep attention layer in the deep self-attention encoder is fixed. In this way, in the training process of the target object detection model, the amount of calculation required for training is reduced, and the obtained weights are guaranteed to be applicable to different three-dimensional holographic data, thereby improving the generalization of the target object detection model, and further, the detection effect of the target object detection model can be improved.

[0279] Furthermore, after obtaining the depth feature vectors to be detected corresponding to each spatial block group to be detected, the electronic device can splice each depth feature vector to be detected according to the position of the spatial block to be detected contained in the corresponding spatial block group to be detected in the slice to which it belongs, to obtain a splicing result to be detected. That is, for a spatial block group to be detected, the position of the spatial block to be detected in the spatial block group to be detected relative to the slice to which it belongs is consistent with the position of the depth feature vector to be detected corresponding to the spatial block group to be detected relative to the splicing result to be detected.

[0280] For example, if a spatial block to be detected in a spatial block group to be detected is located at the first horizontally and first vertically position relative to the slice to which it belongs, then the depth feature vector to be detected corresponding to the spatial block group to be detected is also located at the first horizontally and first vertically position relative to the splicing result to be detected.

[0281] Then, the splicing result to be detected can be input into the spatial self-attention encoder. Accordingly, the spatial self-attention encoder can encode the data corresponding to different areas in the scanning plane (i.e., XOY plane) in the vertical depth direction of the slice, and calculate the correlation between the feature vectors of different areas in the XOY plane (i.e., the depth feature vectors to be detected corresponding to each spatial slice group to be detected). In this way, the spatial self-attention encoder can achieve the effect of global feature extraction and obtain the spatial feature vector to be detected corresponding to the holographic data to be detected.

[0282] The spatial self-attention encoder contains multiple spatial self-attention modules, and each spatial self-attention module is composed of multiple network layers. For example, in a spatial self-attention module, the first layer is the Spatial Self-Attension layer, the second layer is the Add&Norm layer, the third layer is the FeedForward layer, and the fourth layer is the Add&Norm layer.

[0283] In one implementation, the spatial self-attention encoder includes multiple spatial self-attention layers that are connected in series and have different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer.

[0284] Accordingly, step S1052 includes:

[0285] Step 1: Input the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder.

[0286] Step 2: Fuse the output results of each spatial self-attention layer to obtain the spatial feature vector to be detected.

[0287] In an embodiment of the present application, the spatial self-attention encoder includes multiple spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is less than the downsampling multiple of the next spatial self-attention layer. The number of spatial self-attention layers included in the spatial self-attention encoder and the downsampling multiple of each spatial self-attention layer can be set according to actual needs.

[0288] The electronic device can input the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder, and each spatial self-attention layer in the spatial self-attention encoder can output the encoding result of this layer (that is, the output result of the spatial self-attention layer). Furthermore, after obtaining the output results of each spatial self-attention layer, the output results of each spatial self-attention layer can be fused to obtain the spatial feature vector to be detected.

[0289] In one implementation, the output results of each spatial self-attention layer can be fused based on FPN (Feature Pyramid Networks).

[0290] For example, see Figure 8 , Figure 8 A schematic diagram of the structure of a spatial self-attention encoder provided in an embodiment of the present application. Figure 8In , the spatial self-attention encoder includes three serially connected spatial self-attention layers with different downsampling multiples. Accordingly, after obtaining the splicing result to be detected, the splicing result to be detected can be input into PatchEmbed (block embedding) for 4-fold downsampling and block processing. Furthermore, the output result of PatchEmbed can be input into the first spatial self-attention layer (4-fold downsampling) of the spatial self-attention encoder for 4-fold downsampling encoding, the output result of the first spatial self-attention layer can be input into the second spatial self-attention layer (8-fold downsampling) of the spatial self-attention encoder for 8-fold downsampling encoding, and the output result of the second spatial self-attention layer can be input into the third spatial self-attention layer (16-fold downsampling) of the spatial self-attention encoder for 16-fold downsampling encoding. Furthermore, the output result of the first spatial self-attention layer, the output result of the second spatial self-attention layer, and the output result of the third spatial self-attention layer can be input into FPN (feature pyramid network) to obtain the spatial feature vector to be detected.

[0291] Based on the above processing, the encoding results of the splicing results to be detected corresponding to different downsampling multiples can be obtained, that is, the generation of multi-scale feature maps with different resolutions can be achieved. Correspondingly, the fused spatial feature vector to be detected can also retain the characteristics of target objects of different scales (sizes), so that the detection ability of the target object detection model for target objects of different scales (sizes) can be improved.

[0292] With respect to step S106 , detection is performed using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

[0293] In one implementation, the spatial feature vectors of multiple positive sample 3D holographic data and multiple negative sample 3D holographic data can be extracted in the manner provided by the above steps S101 to S105. The positive sample 3D holographic data is obtained by scanning the scanning space where the target object exists through radar; the negative sample 3D holographic data is obtained by scanning the scanning space where the target object does not exist through radar.

[0294] Furthermore, based on the similarity between the spatial feature vectors of multiple positive sample three-dimensional holographic data and the spatial feature vector to be detected, as well as the similarity between the spatial feature vectors of multiple negative sample three-dimensional holographic data and the spatial feature vector to be detected, the detection result representing the target object in the space to be scanned is determined.

[0295] In some embodiments, the electronic device can determine the target objects in the to-be-scanned space through the target object detection model, and when there is at least one target object in the to-be-scanned space, the area occupied by each target object (i.e., the actual position of the target object). Accordingly, the pre-trained target object detection model also includes: a detection head.

[0296] See also Fig. 9 , Fig. 9 A third flow chart of the target object detection method provided in the embodiment of the present application, in Figure 6 On the basis of, step S106 comprises:

[0297] S1061: Input the spatial feature vector to be detected into the detection head to obtain the predicted box and the confidence level corresponding to each predicted box.

[0298] Among them, the confidence corresponding to a prediction box indicates the probability that the prediction box contains the target object.

[0299] S1062: If there is a prediction box whose confidence is greater than a preset confidence threshold, it is determined that there is a target object in the space to be scanned.

[0300] In the embodiment of the present application, the obtained spatial feature vector of the holographic data to be detected can be input into the detection head in the target object detection model. Accordingly, the detection head processes the spatial feature vector to be detected and outputs a prediction box (i.e., the position of the prediction box) and the confidence level corresponding to each prediction box.

[0301] Among them, the confidence corresponding to a prediction box represents the probability that the prediction box contains the target object, and the confidence value is between [0,1].

[0302] That is, the greater the confidence corresponding to a prediction box, the greater the probability that the position corresponding to the prediction box contains the target object. Accordingly, for each prediction box whose confidence is greater than the preset confidence threshold, the electronic device can determine that the position corresponding to the prediction box is: the position of a target object in the space to be scanned. For example, the preset confidence threshold can be 0.6.

[0303] Based on the above processing, the detection head can process the feature vector of the space to be detected to obtain the prediction frame and the confidence level corresponding to each prediction frame. Then, the electronic device can determine the existence of the target object in the space to be scanned according to the confidence level corresponding to each prediction frame. In this way, each target object in the space to be scanned can be accurately located, and the true position of each target object in the space to be scanned can be obtained. In this way, while realizing the detection of the target object, more accurate target object position information can be provided for subsequent processing procedures (such as the positioning of the target object, etc.).

[0304] See also Fig.10 , Fig.10 A schematic diagram of a process for detecting a target object based on three-dimensional holographic data provided in an embodiment of the present application. Fig.10 In the present application, the three-dimensional holographic data (i.e., the holographic data to be detected) contains multiple slices of the same size. The X-axis, Y-axis, and Z-axis respectively represent a dimension of the three-dimensional holographic data. The Z-axis represents the depth dimension of the three-dimensional holographic data, and the Z-axis direction represents the above-mentioned depth direction. The X-axis represents the horizontal spatial dimension of the three-dimensional holographic data, and the X-axis direction represents the above-mentioned horizontal direction. The Y-axis represents the vertical spatial dimension of the three-dimensional holographic data, and the Y-axis direction represents the above-mentioned vertical direction. The X-axis, Y-axis, and Z-axis are perpendicular to each other, and the XOY plane is a scanning plane perpendicular to the depth direction.

[0305] After obtaining the holographic data to be detected, the electronic device can divide each slice in the holographic data to be detected into a plurality of spatial blocks to be detected in the same division manner. Fig.10 In the example, a column corresponding to Z1, a column corresponding to Z2, a column corresponding to Z3, ..., a column corresponding to ZN, respectively correspond to a slice divided into multiple spatial blocks to be detected. Furthermore, multiple spatial blocks to be detected that have the same position in the corresponding slice can be selected from the spatial blocks to be detected obtained by division to obtain a spatial block group to be detected. That is, the spatial blocks to be detected in the same row in Z1, Z2, Z3, ..., ZN can be obtained as a spatial block group to be detected.

[0306] Correspondingly, each spatial block group to be detected can be input into the deep self-attention encoder in the pre-trained target object detection model to obtain the deep feature vector to be detected corresponding to each spatial block group to be detected. Fig.10 In , the deep self-attention encoder contains K self-attention modules. The first layer of each deep self-attention module is the DepthSelf-Attension layer, the second layer is the Add&Norm layer, the third layer is the Feed Forward layer, and the fourth layer is the Add&Norm layer. The first Add&Norm layer (i.e., the second layer) can obtain the original input data of the deep self-attention encoder and the output data of the first layer for processing. The second Add&Norm layer (i.e., the fourth layer) can obtain the output data of the second layer and the output data of the Feed Forward layer for processing. In other words, the model can use the deep self-attention mechanism to learn the relationship between the values ​​of pixels with consistent horizontal and vertical coordinate values ​​in multiple spatial slices to be detected in a group of spatial slices to be detected.

[0307] After obtaining the depth feature vector to be detected corresponding to each spatial block group to be detected, the depth feature vectors to be detected can be spliced ​​according to the position of the spatial block to be detected contained in the corresponding spatial block group to be detected in the slice to which it belongs, to obtain the splicing result to be detected; the splicing result to be detected is input into the spatial self-attention encoder to obtain the spatial feature vector to be detected. The spatial self-attention encoder contains N spatial self-attention modules, the first layer of each spatial self-attention module is the Spatial Self-Attension layer, the second layer is the Add&Norm layer, the third layer is the Feed Forward layer, and the fourth layer is the Add&Norm layer. The first Add&Norm layer (i.e., the second layer) can obtain the original input data of the spatial self-attention encoder and the output data of the first layer for processing. The second Add&Norm layer (i.e., the fourth layer) can obtain the output data of the second layer and the output data of the FeedForward layer for processing. Among them, the model can use the spatial self-attention mechanism to learn the relationship between different spatial slice groups to be detected.

[0308] Each spatial self-attention layer in the spatial self-attention encoder can output the encoding result of this layer. Then, the output results of each spatial self-attention layer can be input into FPN (Feature Pyramid Networks). The feature pyramid network can use the multi-scale structure to fuse the output results of each spatial self-attention layer to obtain the spatial feature vector to be detected. Then, the spatial feature vector to be detected can be input into the detection head (i.e., 2Dhead) to obtain the prediction box and the confidence corresponding to each prediction box.

[0309] In this application, the electromagnetic waves emitted by the radar are millimeter waves, and the penetration of millimeter waves is not high. For example, millimeter waves usually cannot penetrate the human body, metal and other media. Therefore, if the three-dimensional holographic data is constructed only based on the scanning results obtained by the radar scanning the space to be scanned at a single angle, it may cause the information of a part of the position in the space to be scanned to be missing in the obtained three-dimensional holographic data, affecting the accuracy of the target object detection.

[0310] In the present application, in the process of scanning the space to be scanned by radar, the electronic device can obtain the scanning results (i.e., echo information) at multiple angles. Accordingly, the target object detection method provided in the present application can combine the scanning results at multiple angles to obtain three-dimensional holographic data representing the space to be scanned (i.e., holographic data to be detected).

[0311] In some embodiments, see Fig.11 , Fig.11 A flow chart of obtaining holographic data to be detected is provided in an embodiment of the present application. The steps include:

[0312] S1101: For each preset scanning angle, acquire echo information obtained by the radar when scanning the space to be scanned at the preset scanning angle as first echo information to be detected.

[0313] Among them, the first echo information to be detected under a preset scanning angle includes: the echo signal at each observation unit obtained by scanning under the preset scanning angle; the multiple preset scanning angles include: multiple preset front scanning angles and multiple preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the space to be scanned; the preset back scanning angle represents the scanning angle when scanning the other side of the space to be scanned.

[0314] S1102: The first echo information to be detected obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second echo information to be detected at each preset front scanning angle; and the first echo information to be detected obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second echo information to be detected at each preset back scanning angle.

[0315] The difference between the specified front scanning angle and the specified back scanning angle is 180°.

[0316] S1103: coherently synthesize the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and coherently synthesize the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected.

[0317] S1104: Based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are merged to obtain the omnidirectional echo information to be detected.

[0318] The preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle.

[0319] S1105: Processing the omnidirectional echo information to be detected based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as the holographic data to be detected.

[0320] In the embodiment of the present application, the echo information obtained by the radar when scanning the space to be scanned at multiple preset scanning angles includes information of each real position in the space to be scanned.

[0321] It can be understood that in the above-mentioned scenario of security inspection of the scanned object in the scanning space, if the scanned object is a person or a metal object, due to the limitation of millimeter wave penetration, the scanning result of the radar at any angle does not include information on the opposite side (the other side of the scanning object where the radar is located) (that is, the radar does not have the ability to observe the opposite side).

[0322] Therefore, the multiple preset scanning angles can be divided into multiple preset front scanning angles and multiple preset back scanning angles. The preset front scanning angle represents the scanning angle when the radar scans one side of the space to be scanned; the preset back scanning angle represents the scanning angle when scanning the other side of the space to be scanned. Subsequently, the multiple preset front scanning angles and the multiple preset back scanning angles are processed independently. The specific processing process for the multiple preset front scanning angles and the processing process for the multiple preset back scanning angles will be described later.

[0323] For each preset scanning angle, the radar can emit electromagnetic waves at the preset scanning angle to obtain an echo signal at each observation unit as the first echo information to be detected at the preset scanning angle.

[0324] Among them, the range of preset scanning angle is The echo signal at an observation unit in the first echo information to be detected under a preset scanning angle can be expressed as: . Where θ represents the preset scanning angle, ( ) Represents the horizontal coordinate value of an observation unit, ( ) It represents the vertical coordinate value of an observation unit, and t represents the time.

[0325] It can be understood that for any preset scanning angle, when the radar scans at the preset scanning angle, the position of each observation unit in the first echo information to be detected obtained represents: the relative position between the observation unit and the current position of the radar. In other words, for a real position in the space to be scanned, when the radar scans at different preset scanning angles, the horizontal coordinate value, the vertical coordinate value, and the time interval of the observation unit corresponding to the same real position in the first echo information to be detected obtained at different preset scanning angles may be different.

[0326] In order to ensure the uniqueness of the horizontal coordinate value, the vertical coordinate value, and the time interval used to characterize the same real position when subsequently describing the information of the same real position in the scanned space, the electronic device can adjust the first echo information to be detected at each preset scanning angle. The adjustment process will be described in subsequent embodiments.

[0327] Accordingly, for multiple preset frontal scanning angles, the first echo information to be detected obtained at each preset frontal scanning angle can be adjusted to the same specified frontal scanning angle to obtain the second echo information to be detected at each preset frontal scanning angle. Accordingly, the coordinate system with the current position as the origin constructed by the radar when scanning at the specified frontal scanning angle can be called the frontal standard coordinate system. For each observation unit in the second echo information to be detected at each preset frontal scanning angle, the horizontal coordinate value, vertical coordinate value and time interval of the observation unit in the frontal standard coordinate system can be used to represent it. For example, the specified frontal scanning angle can be: a preset frontal scanning angle selected from multiple preset frontal scanning angles.

[0328] For multiple preset back side scanning angles, the first echo information to be detected obtained at each preset back side scanning angle can be adjusted to the same specified back side scanning angle to obtain the second echo information to be detected at each preset back side scanning angle. Correspondingly, the coordinate system with the current position as the origin constructed by the radar when scanning at the specified back side scanning angle can be called the back side standard coordinate system. For each observation unit in the second echo information to be detected at each preset back side scanning angle, the horizontal coordinate value, vertical coordinate value and time interval of the observation unit in the back side standard coordinate system can be used to represent it. For example, the specified back side scanning angle can be: a preset back side scanning angle selected from multiple preset back side scanning angles.

[0329] It is understandable that the first echo information to be detected obtained at the specified front scanning angle and the first echo information to be detected obtained at the specified back scanning angle may not be adjusted. The specified front scanning angle differs from the specified back scanning angle by 180°.

[0330] Scanning angle means: the angle between the scanning direction of the radar and the specified vertical plane in the space to be scanned.

[0331] See also Fig.12 , Fig.12 A schematic top view of a scene in which a radar is scanned at various preset scanning angles provided in an embodiment of the present application. Fig.12In order to facilitate description, the specified vertical plane in the space to be scanned can be the plane perpendicular to the ground corresponding to the front of the human body at a bird's-eye view. Accordingly, when the radar is located on the extension line of the normal line of the front of the human body, the scanning angle of the radar is 90°. Accordingly, the preset scanning angle in the range of 0° to 180° is the preset front scanning angle, and the preset scanning angle in the range of 180° to 360° is the preset back scanning angle. The white-filled circle indicates the position of the radar when scanning at a preset front scanning angle, and the black-filled circle indicates the position of the radar when scanning at a preset back scanning angle. The rectangle marked with a human body looking down indicates the position of the scanning object (person) in the space to be scanned from a bird's-eye view. The first echo information to be detected by the radar at each preset front scanning angle belongs to the front radar echo signal group, and the first echo information to be detected by the radar at each preset back scanning angle belongs to the back radar echo signal group. In addition, Fig.12 In the figure, multiple preset front scanning angles can be divided into 18°, 54°, 90°, 126°, and 162°, and multiple preset back scanning angles can be divided into 198°, 234°, 270°, 306°, and 342°, arranged counterclockwise from the direction indicated by 0°. Among them, the front scanning angle is specified to be 90°, and the coordinate system with the current position as the origin constructed by the radar when scanning at the specified front scanning angle can be called the front standard coordinate system. The back scanning angle is specified to be 270° (-90°), and the coordinate system with the current position as the origin constructed by the radar when scanning at the specified back scanning angle can be called the back standard coordinate system.

[0332] In some embodiments, step S1102 includes: adjusting the first echo information to be detected obtained at each preset scanning angle according to a first formula to obtain the second echo information to be detected at each preset scanning angle.

[0333] Among them, the first formula is:

[0334] ;

[0335] θ represents a preset scanning angle, Indicates the observation unit in the first echo information to be detected under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second echo information to be detected under the preset scanning angle; Represents a collection; Represents a natural constant.

[0336] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle.

[0337] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle.

[0338] In an embodiment of the present application, a standard part can be placed in advance in the space to be scanned, and for a real position in the space to be scanned (i.e., a position on the surface of the standard part), the horizontal coordinate value (which can be called the first horizontal coordinate value), the vertical coordinate value (which can be called the first vertical coordinate value) of the observation unit corresponding to the real position when the radar is scanning at a preset frontal scanning angle, and the amplitude (which can be called the first amplitude) and phase (which can be called the first phase) of the echo signal at the observation unit can be obtained in advance. And the horizontal coordinate value (which can be called the second horizontal coordinate value), the vertical coordinate value (which can be called the second vertical coordinate value) of the observation unit corresponding to the real position when the radar is scanning at a specified frontal scanning angle, and the amplitude (which can be called the second amplitude) and phase (which can be called the second phase) of the echo signal at the observation unit can be obtained in advance.

[0339] Furthermore, the difference between the first transverse coordinate value and the second transverse coordinate value is calculated as the transverse position compensation of the observation unit at the preset frontal scanning angle; the difference between the first longitudinal coordinate value and the second longitudinal coordinate value is calculated as the longitudinal position compensation of the observation unit at the preset frontal scanning angle; the difference between the first amplitude and the second amplitude is calculated as the amplitude compensation of the observation unit at the preset frontal scanning angle; the difference between the first phase and the second phase is calculated as the phase compensation of the observation unit at the preset frontal scanning angle.

[0340] Similarly, for a real position in the space to be scanned, the horizontal coordinate value (which may be called the third horizontal coordinate value) and the vertical coordinate value (which may be called the third vertical coordinate value) of the observation unit corresponding to the real position when the radar is scanning at a preset back scanning angle, as well as the amplitude (which may be called the third amplitude) and the phase (which may be called the third phase) of the echo signal at the observation unit can be obtained in advance. And the horizontal coordinate value (which may be called the fourth horizontal coordinate value) and the vertical coordinate value (which may be called the fourth vertical coordinate value) of the observation unit corresponding to the real position when the radar is scanning at a specified back scanning angle can be obtained in advance.

[0341] Furthermore, the difference between the third transverse coordinate value and the fourth transverse coordinate value is calculated as the transverse position compensation amount of the observation unit at the preset back scanning angle; the difference between the third longitudinal coordinate value and the fourth longitudinal coordinate value is calculated as the longitudinal position compensation amount of the observation unit at the preset back scanning angle; the difference between the third amplitude and the fourth amplitude is calculated as the amplitude compensation amount of the observation unit at the preset back scanning angle; the difference between the third phase and the fourth phase is calculated as the phase compensation amount of the observation unit at the preset back scanning angle.

[0342] Correspondingly, for the first echo information to be detected obtained at each preset scanning angle, the electronic device can adjust the first echo information to be detected according to the first formula to obtain the second echo information to be detected at each preset scanning angle.

[0343] Based on the above processing, for a real position of the surface of the standard part in the space to be scanned, the echo signal of the observation unit representing the real position obtained by scanning the radar at a preset front scanning angle, and the echo signal of the observation unit representing the real position obtained by scanning the radar at a specified front scanning angle can be obtained in advance, and then, according to the difference between the two echo signals, the phase compensation amount of the observation unit at the preset front scanning angle is determined. Similarly, the phase compensation amount of each observation unit at each preset back scanning angle is determined. It is also possible to adjust the first echo information to be detected obtained at each preset front scanning angle to the same specified front scanning angle, and adjust the first echo information to be detected obtained at each preset back scanning angle to the same specified back scanning angle, so as to obtain complete three-dimensional holographic data.

[0344] With respect to step S1103, after obtaining the second echo information to be detected at each preset frontal scanning angle, since the second echo information to be detected at two adjacent preset frontal scanning angles may contain echo signals at the observation unit representing the same real position, that is, the second echo information to be detected at multiple preset frontal scanning angles may contain repeated information for the same real position, the electronic device may perform coherent synthesis on the second echo information to be detected at each preset frontal scanning angle to obtain the frontal echo information to be detected.

[0345] Similarly, the electronic device can perform coherent synthesis on the second echo information to be detected at each preset back side scanning angle to obtain the back side echo information to be detected.

[0346] In some embodiments, step S1103 includes: based on the second formula, coherently synthesizing the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and coherently synthesizing the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected.

[0347] Among them, the second formula is:

[0348] ;

[0349] ;

[0350] Where θ represents a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second echo information to be detected under the preset scanning angle; Indicates the positive echo information to be detected. Indicates the back echo information to be detected; Indicates sum.

[0351] In the embodiment of the present application, for the second echo information to be detected at each preset frontal scanning angle, the second formula can be used as follows: The coherent synthesis calculation is performed on the part of the back side to obtain the front echo information to be detected. For the second echo information to be detected under each preset back side scanning angle, it can be based on the second formula The coherent synthesis calculation is performed on the part to obtain the back echo information to be detected.

[0352] Correspondingly, for a real position in the space to be scanned, the echo signals at each observation unit corresponding to the real position in the second echo information to be detected under each preset frontal scanning angle can be coherently synthesized according to the second formula mentioned above to obtain the coherent synthesis result of the real position in the space to be scanned (i.e., the frontal echo information to be detected).

[0353] Similarly, for a real position in the space to be scanned, the echo signals at each observation unit corresponding to the real position in the second echo information to be detected at each preset back scanning angle can be coherently synthesized according to the second formula mentioned above to obtain the coherent synthesis result of the real position in the space to be scanned (i.e., the back echo information to be detected).

[0354] Since the second echo information to be detected at each preset frontal scanning angle is: the first echo information to be detected obtained at each preset frontal scanning angle is adjusted to the first echo information obtained at the same specified frontal scanning angle. Therefore, the observation units with the same horizontal coordinate value and vertical coordinate value in the second echo information to be detected at each preset frontal scanning angle represent the same real position. Accordingly, in the process of coherently synthesizing the second echo information to be detected at each preset frontal scanning angle, the corrected echo signals at the observation units with the same horizontal coordinate value and the same vertical coordinate value can be directly superimposed.

[0355] Similarly, in the process of coherently synthesizing the second echo information to be detected at each preset back scanning angle, the corrected echo signals at the observation units with the same transverse coordinate value and the same longitudinal coordinate value can be directly superimposed.

[0356] According to the physical principle of electromagnetic wave coherence enhancement, for the same real position, if the echo signal corresponding to the real position is accurate, the echo signal of the real position is coherently synthesized to enhance the amplitude of the echo signal. Therefore, based on the above processing, the proportion of effective information in the front echo information to be detected and the back echo information to be detected can be increased. In addition, in the process of coherently synthesizing the second echo information to be detected at each preset front scanning angle, since the noise in each echo signal is random and the value of the noise is usually small, the second echo information to be detected at each preset front scanning angle is coherently synthesized, which can reduce the influence of noise in the front echo information to be detected. Similarly, the second echo information to be detected at each preset back scanning angle is coherently synthesized, which can reduce the influence of noise in the back echo information to be detected.

[0357] In this way, the duplication of information between the second echo information to be detected can be reduced, while the proportion of effective information in the front echo information to be detected and the back echo information to be detected can be increased, and the influence of noise in the front echo information to be detected and the back echo information to be detected can be reduced. In addition, the accuracy of the subsequently obtained three-dimensional holographic data can be improved, which can also improve the accuracy of subsequent target detection.

[0358] Furthermore, after obtaining the front echo information to be detected and the back echo information to be detected, since the coordinate system of the front echo information to be detected is: the front standard coordinate system with the current position as the origin constructed when the radar scans at a specified front scanning angle. Correspondingly, the coordinate system of the back echo information to be detected is: the back standard coordinate system with the current position as the origin constructed when the radar scans at a specified back scanning angle.

[0359] Therefore, it is necessary to unify the front echo information to be detected and the back echo information to be detected in the same coordinate system. That is, the electronic device can fuse the front echo information to be detected and the back echo information to be detected based on the preset observation time and time anti-folding processing to obtain the fusion result of the front echo information to be detected and the back echo information to be detected (that is, the omnidirectional echo information to be detected).

[0360] In some embodiments, step S1104 includes: using the third formula, based on the preset observation time and time deconvolution processing, fusing the front echo information to be detected and the back echo information to be detected to obtain the omnidirectional echo information to be detected.

[0361] Among them, the third formula is:

[0362] ;

[0363] Indicates one of the front echo information to be detected and the back echo information to be detected; represents the result obtained by processing the other of the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution; T represents the preset observation time; t represents time; Indicates the echo information to be detected in all directions.

[0364] In the embodiment of the present application, the preset observation time means: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle. It indicates that time deconvolution processing is performed on one of the front echo information to be detected and the back echo information to be detected based on the preset observation time.

[0365] That is to say, the omnidirectional echo information to be detected is obtained by linearly superimposing the time deconvolution results corresponding to the front echo information to be detected and the back echo information to be detected. Alternatively, the omnidirectional echo information to be detected is obtained by linearly superimposing the time deconvolution results corresponding to the back echo information to be detected and the front echo information to be detected.

[0366] In the process of linear superposition, the echo signals at the observation units with the same horizontal coordinate value and the same vertical coordinate value in the time deconvolution results corresponding to the front echo information to be detected and the back echo information to be detected can be directly superimposed. Alternatively, in the process of linear superposition, the echo signals at the observation units with the same horizontal coordinate value and the same vertical coordinate value in the time deconvolution results corresponding to the back echo information to be detected and the front echo information to be detected can be directly superimposed.

[0367] In this way, the front echo information to be detected and the back echo information to be detected can be unified into the same coordinate system to obtain the omnidirectional echo information to be detected. In other words, the omnidirectional echo information to be detected can represent the complete information collected at multiple angles in the space to be scanned. Subsequently, the holographic data to be detected is obtained based on the omnidirectional echo information to be detected, which can fully represent the information of each real position in the space to be scanned and improve the accuracy of subsequent target detection.

[0368] For S1105, the electronic device can process the omnidirectional echo information to be detected based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as the holographic data to be detected. The radar holographic imaging algorithm can be: RMA (range migration) algorithm, BP (BackProjection) algorithm, etc.

[0369] Based on the above processing, the echo signals obtained at each preset scanning angle can be losslessly fused to obtain complete three-dimensional holographic data for the space to be scanned. Accordingly, the three-dimensional holographic data can fully represent the surface information of the scanned object in the space to be scanned. Subsequently, target object detection can be performed based on the three-dimensional holographic data to ensure the accuracy of detection.

[0370] Moreover, during the above processing, the electronic device does not need to perform pre-processing such as cropping and filtering on the echo signal, which can also improve the accuracy of the three-dimensional holographic data and reduce noise interference. In this way, the information loss caused by pre-processing can be avoided, and the accuracy and resolution of target object detection can be further improved.

[0371] See also Fig.13 , Fig.13 A schematic diagram of a process for obtaining holographic data to be detected provided in an embodiment of the present application. Fig.13In the embodiment, the electronic device can obtain the first echo information to be detected under each preset scanning angle, that is, the radar echo signal of angle 1, the radar echo signal of angle 2, ..., and the radar echo signal of angle N. Among them, some angles of angle 1, angle 2, ..., and angle N belong to the preset front scanning angle, and the other angles belong to the preset back scanning angle. Furthermore, the electronic device can adjust the first echo information to be detected obtained under each preset front scanning angle to the same specified front scanning angle, and obtain the second echo information to be detected under each preset front scanning angle. The second echo information to be detected under each preset front scanning angle is coherently synthesized to obtain the front echo information to be detected. That is, the forward radar echo signal is compensated and synthesized. In addition, the first echo information to be detected obtained under each preset back scanning angle can be adjusted to the same specified back scanning angle, and the second echo information to be detected under each preset back scanning angle is obtained. The second echo information to be detected under each preset back scanning angle is coherently synthesized to obtain the back echo information to be detected. That is, the back radar echo signal is compensated and synthesized.

[0372] Furthermore, the electronic device can fuse the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution processing to obtain the omnidirectional echo information to be detected. That is, the front and back radar echoes are synthesized. The omnidirectional echo information to be detected is processed based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as the holographic data to be detected. That is, holographic imaging is performed to obtain millimeter-wave omnidirectional holographic data.

[0373] Based on the same inventive concept, an embodiment of the present application provides a model training method, which can be applied to electronic devices.

[0374] See also Fig.14 , Fig.14 The first flow chart of the model training method provided in the embodiment of the present application comprises the following steps:

[0375] S1401: Obtain sample holographic data of the current batch and labels of each sample holographic data.

[0376] Among them, the label of a sample holographic data represents the true value box in the sample holographic data, and each true value box represents the area occupied by a target object; a sample holographic data contains multiple slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, and the value of the pixel point contains the characteristic information of the echo signal obtained by scanning the real position, and the scanning depth of the real position corresponding to each pixel point in a slice is the same.

[0377] S1402: For each sample holographic data of the current batch, each slice in the sample holographic data is divided into a plurality of sample space blocks according to the same division method.

[0378] S1403: Select multiple sample space blocks with consistent positions in the corresponding slices from the sample space blocks obtained by division to obtain a sample space block group.

[0379] S1404: Input each sample space block group into the deep self-attention encoder in the target object detection model in the initial state to obtain the sample depth feature vector corresponding to each sample space block group.

[0380] Among them, the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head.

[0381] S1405: According to the positions of the sample space blocks included in the corresponding sample space block group in the corresponding slice, the depth feature vectors of each sample are spliced ​​to obtain a sample splicing result.

[0382] S1406: Input the sample concatenation result into the spatial self-attention encoder to obtain the sample space feature vector.

[0383] S1407: Input the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data.

[0384] S1408: Calculate the loss value based on the prediction result of the sample holographic data and the difference between the label corresponding to the sample holographic data.

[0385] S1409: Based on the obtained loss value, adjust the model parameters of the target object detection model in the initial state.

[0386] Based on the above processing, the trained target object detection model can be combined with the characteristics of three-dimensional holographic data (the data redundancy of three-dimensional holographic data in the depth dimension is high), and based on the deep self-attention encoder, the data corresponding to each spatial block group to be detected can be compressed in the depth dimension, and the invalid redundant information in the three-dimensional holographic data can be filtered. It is also possible to reduce the amount of calculation of model processing and improve the efficiency of model processing in the subsequent processing flow. Then, the splicing result to be detected can be input into the spatial self-attention encoder. Correspondingly, the spatial self-attention encoder can encode the data corresponding to different regions in the scanning plane (i.e., the XOY plane) in the vertical depth direction of the slice, and calculate the correlation between the feature vectors of different regions in the XOY plane (i.e., the depth feature vector to be detected corresponding to each spatial block group to be detected). In this way, the spatial self-attention encoder can achieve the effect of global feature extraction and obtain the spatial feature vector to be detected corresponding to the holographic data to be detected. Then, the detection head can process the spatial feature vector to be detected to obtain the prediction frame and the confidence corresponding to each prediction frame. Then, the electronic device can determine the existence of the target object in the space to be scanned according to the confidence corresponding to each prediction frame. In this way, each target object in the space to be scanned can be accurately located, and the real position of each target object in the space to be scanned can be obtained.

[0387] Subsequently, after obtaining three-dimensional holographic data (i.e., holographic data to be detected) by scanning the space to be scanned by radar, the electronic device can automatically process the holographic data to be detected based on the trained target object detection model to obtain the detection result of the holographic data to be detected, thus realizing automatic detection and improving the efficiency of detection. Furthermore, it can provide more accurate location information of the target object while realizing the detection of the target object.

[0388] In addition, the target object detection model in this application is: a large millimeter-wave holographic model based on the Transformer architecture. Correspondingly, the target object detection model can utilize the global feature extraction capability of the Transformer architecture to enhance the model's learning of holographic data. On this basis, a processing method combining the deep self-attention module and the spatial self-attention module is proposed, which can save computational effort when processing dense point clouds (i.e., three-dimensional holographic data) and improve the convergence speed of the model during training.

[0389] With respect to step S1401, the sample holographic data of the current batch and the label of each sample holographic data are obtained. The process of selecting the sample holographic data of the current batch will be described in the subsequent embodiments.

[0390] Among them, one sample holographic data includes multiple slices of the same size. For the description of the slices, reference may be made to the relevant description in step S101, which will not be repeated here.

[0391] The label of a sample holographic data represents a true value box in the sample holographic data, and each true value box represents an area occupied by a target object.

[0392] For step S1402, for each sample holographic data of the current batch, each slice in the sample holographic data is divided into multiple sample space blocks according to the same division method. The process of obtaining multiple sample space blocks of each slice can refer to the process of obtaining multiple to-be-detected space blocks of each slice in the above step S102, which will not be described in detail here.

[0393] For step S1403, multiple sample space blocks with consistent positions in the corresponding slices are selected from the divided sample space blocks to obtain a sample space block group. The process of obtaining each sample space block group can refer to the process of obtaining each space block group to be detected in the above step S103, which will not be repeated here.

[0394] For step S1404, each sample space block group is respectively input into the deep self-attention encoder in the target object detection model in the initial state to obtain the sample depth feature vector corresponding to each sample space block group. Among them, the description of the deep self-attention encoder in the target object detection model can refer to the relevant description in the above step S104, which will not be repeated here. The process of obtaining the sample depth feature vector corresponding to each sample space block group can refer to the process of obtaining the depth feature vector to be detected corresponding to each space block group to be detected in the above step S104, which will not be repeated here.

[0395] For step S1405, the depth feature vectors of each sample are spliced ​​according to the position of the sample space blocks contained in the corresponding sample space block group in the corresponding slice to obtain a sample splicing result. The process of obtaining the sample splicing result can refer to the process of obtaining the splicing result to be detected in the above step S1051, which will not be repeated here.

[0396] For step S1406, the sample concatenation result is input into the spatial self-attention encoder to obtain the sample space feature vector. The process of obtaining the sample space feature vector can refer to the process of obtaining the space feature vector to be detected in the above step S1052, which will not be repeated here.

[0397] For step S1407, the sample space feature vector is input to the detection head to obtain the prediction result of the sample holographic data. The process of obtaining the prediction result of the sample holographic data can refer to the process of obtaining the detection result of the holographic data to be detected in the above step S106, which will not be repeated here.

[0398] For step S1408 to step S1409, the loss value is calculated based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data. The process of calculating the loss value will be described in subsequent embodiments. Accordingly, the electronic device can adjust the model parameters of the target object detection model in the initial state based on the obtained loss value.

[0399] In one implementation, the target object detection model training may be terminated when the model parameters of the target object detection model reach convergence. Alternatively, the target object detection model training may be terminated when a specified number of training times is reached.

[0400] The target object detection model in the initial state may be a target object detection model under the original model parameters (i.e., a target object detection model that has not been pre-trained). Alternatively, the target object detection model in the initial state may also be obtained by pre-training the target object detection model under the original model parameters.

[0401] In some embodiments, the sample holographic data of the current batch is selected from the full amount of sample holographic data. The target object detection model in the initial state is obtained by pre-training based on the high-quality sample holographic data and the label corresponding to each high-quality sample holographic data. The image blurriness of the high-quality sample holographic data is lower than the image blurriness of other sample holographic data in the full amount of sample holographic data.

[0402] See also Fig.15 , Fig.15 This is an example diagram of a visualization result obtained by visualizing three-dimensional holographic data provided in an embodiment of the present application. In the above security inspection scenario, Fig.15 In the image, there is a target object (i.e., a lighter) in the area corresponding to the rectangular box in the left image, and there is no target object in the area corresponding to the rectangular box in the right image. However, due to the wrinkles in the clothes worn by the scanned object (person), the information of the real position of the lighter in the 3D holographic data corresponding to the left image may be close to the information of the real position of the wrinkles in the 3D holographic data corresponding to the right image. That is, the visualization result of the 3D holographic data with the target object is difficult to distinguish from the visualization result of the 3D holographic data without the target object. In other words, the 3D holographic data corresponding to the left image is easily judged as not having the target object, which leads to misjudgment.

[0403] It can be seen that in actual application scenarios, due to the fuzzy and difficult-to-distinguish characteristics of millimeter wave data, if the full amount of sample holographic data is directly used for training, the learning direction of the model parameters may be unclear and the model parameters may be difficult to converge.

[0404] Therefore, in the embodiment of the present application, before training the target object detection model in the initial state according to the above steps S1401 to S1409, the target object detection model in the initial state can be pre-trained based on the high-quality sample holographic data and the labels corresponding to each high-quality sample holographic data. The process of pre-training the target object detection model in the initial state based on the high-quality sample holographic data and the labels corresponding to each high-quality sample holographic data can refer to the process of training based on the current batch of sample holographic data and the labels of each sample holographic data in steps S1401 to S1409, which will not be described in detail here.

[0405] The high-quality sample holographic data can be selected by the technician in advance from the full amount of sample holographic data. For example, the technician can select the sample holographic data containing the target object that is easy to distinguish from the full amount of sample holographic data as the high-quality sample holographic data, and the rest can be called the difficult-to-distinguish sample holographic data. The image blur degree of the high-quality sample holographic data is lower than the image blur degree of other sample holographic data in the full amount of sample holographic data.

[0406] In one implementation, after obtaining the full amount of sample holographic data, the electronic device can perform visualization processing on each sample holographic data to obtain the visualization results of each sample holographic data. Accordingly, the technician can select multiple high-quality sample holographic data from the full amount of sample holographic data according to the visualization results of each sample holographic data.

[0407] For each sample holographic data, the maximum coordinate value of each pixel in the sample holographic data in the depth direction can be obtained, and image normalization processing can be performed, and the obtained processing result is used as the visualization result of the sample holographic data.

[0408] If the visualization result of a sample holographic data is as above Fig.15 From the left image shown in , it can be determined that the sample holographic data is difficult-to-distinguish sample holographic data.

[0409] Based on the above processing, during the training process of the target object detection model, a portion of high-quality sample holographic data can be pre-selected for pre-training to obtain the target object detection model in the initial state. That is, a small amount of high-quality and easy-to-separate training data (high-quality sample holographic data) can be selected as the starting data, and the training model can obtain the pre-training weights. Accordingly, in the subsequent training process, the pre-training weights can be used to train based on the full amount of training data to obtain the pre-frame (i.e., the forward candidate frame) of each sample holographic data in the full amount of training data.

[0410] Accordingly, in the process of training the target object detection model in the initial state according to the above steps S1401 to S1409, the electronic device can randomly select a preset number of sample holographic data from the full amount of sample holographic data as the sample holographic data of the current batch. The sample holographic data of the current batch can be all the sample holographic data in the full amount of sample holographic data, or can be a part of the sample holographic data in the full amount of sample holographic data.

[0411] Furthermore, after each sample holographic data in the current batch of sample holographic data is used for training, a second preset number of sample holographic data may be reselected as the sample holographic data of the current batch.

[0412] Based on the above processing, during the training process of the target object detection model, the current batch of sample holographic data can be selected from the full amount of sample holographic data for training, so as to realize batch training of the target object detection model. In this way, it is possible to combine the fuzzy characteristics of three-dimensional holographic data and propose a stage learning strategy based on millimeter wave feature fuzziness. By performing stage learning on feature fuzziness, the model convergence speed is accelerated, and the model resolution and generalization capabilities are improved.

[0413] In some embodiments, the prediction result of a sample holographic data includes: a prediction box in the sample holographic data, and a confidence level corresponding to each prediction box; the confidence level corresponding to a prediction box indicates the probability that the prediction box contains a target object.

[0414] Step S1408 includes:

[0415] Step 1: For each truth frame in the label of the sample holographic data, calculate the fuzziness of the truth frame.

[0416] Among them, the fuzziness of a true value box represents: the detection difficulty of the current target object detection model for the target object in the true value box.

[0417] Step 2: Calculate the loss value using the position difference between a true value frame whose ambiguity is not greater than the current ambiguity threshold and a predicted frame corresponding to the true value frame, as well as the confidence corresponding to the predicted frame.

[0418] In the embodiment of the present application, the detection head of the target object detection model can obtain the prediction result of the sample holographic data according to the sample space feature vector of the input sample holographic data. The prediction result of a sample holographic data includes: the prediction box in the sample holographic data, and the confidence corresponding to each prediction box; the confidence corresponding to a prediction box indicates the probability that the prediction box contains the target object.

[0419] Correspondingly, for each truth frame in the label of the sample holographic data, the electronic device can calculate the ambiguity of the truth frame based on the truth frame and each prediction frame.

[0420] In one implementation, the above step 1 includes:

[0421] Step a: For each true value frame in the label of the sample holographic data, calculate the intersection-over-union ratio between the true value frame and each predicted frame respectively.

[0422] Step b: Calculate the product of each IoU ratio and the confidence of the prediction box used to obtain the IoU ratio.

[0423] Step c: The inverse of the maximum value in each product is determined as the fuzziness of the true value frame.

[0424] In the embodiment of the present application, the fuzziness of a true value frame can be calculated based on the seventh formula. The seventh formula is:

[0425] ;

[0426] ;

[0427] in, is a truth box, is the i-th prediction box among all prediction boxes output by the target object detection model, is the confidence corresponding to the i-th prediction box, is the intersection-and-ratio function.

[0428] It can be understood that the clearer the target object represented by a true value frame is, that is, the easier it is to distinguish the target object in the three-dimensional hologram, and accordingly, the position of the prediction frame representing the target object obtained based on the target object detection model is usually closer to the true value frame (that is, the intersection and union ratio is closer to 1), and the confidence of the prediction frame is greater (that is, the closer to 1). The larger the value of is, the fuzziness of the truth box ( ) is smaller (closer to 1).

[0429] Therefore, the smaller the ambiguity of a true value box, the easier it is to distinguish the target object represented by the true value box relative to the current target object detection model.

[0430] Based on the above processing, for each truth frame in a sample holographic data, the fuzziness of the truth frame can be calculated, that is, the detection difficulty of the target object represented by the truth frame can be quantified. Accordingly, at different stages of training, the truth frames of target objects with different detection difficulties can be selected to calculate the loss value, so as to realize the gradient training of the target object detection model.

[0431] With respect to the above step 2, for each true value frame in the label of the sample holographic data, a prediction frame corresponding to the true value frame is: the prediction frame used to calculate the fuzziness of the true value frame.

[0432] Correspondingly, if the fuzziness of the true value frame is not greater than the current fuzziness threshold, the first sub-loss value can be calculated based on the position difference between the true value frame and the prediction frame corresponding to the true value frame. And based on the confidence corresponding to the prediction frame, the second sub-loss value is calculated. Furthermore, the loss value of the true value frame and the prediction frame can be calculated based on the first sub-loss value and the second sub-loss value. For example, the weighted sum of the first sub-loss value and the second sub-loss value can be calculated as the loss value of the true value frame and the prediction frame. Furthermore, based on the loss value of the true value frame and the prediction frame, the model parameters of the current target object detection model can be adjusted.

[0433] If the fuzziness of the truth box is greater than the current fuzziness threshold, it indicates that the detection difficulty of the target object represented by the truth box is higher than that of the current target object detection model. Accordingly, the truth box is not used to calculate the loss value.

[0434] Based on the above processing, for each truth frame in the label of the sample holographic data, the truth frame with blur greater than the set threshold can be regarded as an ignored area, and does not participate in the loss calculation and return; the truth frame with blur not greater than the set threshold is regarded as a positive sample area, and the loss is calculated normally and returned. That is, in the training stage, the truth frame corresponding to the target object with lower detection difficulty can be selected to calculate the loss value. Later, after the detection effect of the target object detection model is improved, the truth frame corresponding to the target object with higher detection difficulty is selected to calculate the loss value, so as to realize the gradient training of the target object detection model.

[0435] In some embodiments, the method further includes: after adjusting the model parameters of the target object detection model in the initial state based on the loss value obtained from the current batch of sample holographic data, increasing the current blur threshold.

[0436] In the embodiment of the present application, after adjusting the model parameters of the target object detection model based on a batch of sample holographic data, the detection capability of the adjusted target object detection model for the target object is usually enhanced, and accordingly, the current fuzziness threshold can be increased, that is, in the process of training the current target object detection model based on the next batch of sample holographic data, the true value box corresponding to the target object with higher detection difficulty can be used to calculate the loss value and adjust the model parameters. For example, the fuzziness threshold initially set can be 2.

[0437] Based on the above processing, during the training stage of the target object detection model, the fuzziness threshold can be gradually increased. That is, in the early stage of training, the target object detection model mainly learns the features of target objects with lower detection difficulty (i.e., simple samples with small fuzziness). In the later stage of training, the features of target objects with higher detection difficulty (i.e., difficult samples with large fuzziness) are learned. In this way, the convergence speed of the model can be improved, and the target object detection model can learn the main features more accurately, reducing the overfitting of interference features.

[0438] It is understandable that, for the same target object, as the model parameters of the target object detection model are continuously optimized during the training process, the fuzziness of the true value box to which the target object belongs will also change (i.e., decrease).

[0439] See also Fig.16 , Fig.16 An example flow chart of a target object detection model training process provided in this application. Fig.16 In the present invention, the electronic device can perform pre-training based on the high-quality sample holographic data (i.e., high-quality easily distinguishable training data) and the labels corresponding to each high-quality sample holographic data to obtain the target object detection model (i.e., model) in the initial state. At this time, the weight of the target object detection model in the initial state can be called the pre-training weight.

[0440] Furthermore, the initial state target object detection model can be trained based on the full amount of training data (i.e., the full amount of sample holographic data), and based on forward processing, the candidate frame (i.e., the predicted frame) of each sample holographic data can be obtained. Accordingly, for each true value frame (i.e., the true value) in the label of the sample holographic data, the fuzziness of the true value frame can be calculated. Furthermore, the fuzziness greater than the fuzziness threshold (i.e., fuzziness> ) is regarded as the ignored area and does not participate in the loss calculation and return. ) is used as the true value box of the positive sample area, and the loss is calculated normally and sent back.

[0441] Among them, during the model training process, the fuzziness threshold can be gradually increased. represents the minimum value of the blur threshold, Indicates the maximum value of the blur threshold.

[0442] In some embodiments, the spatial self-attention encoder includes multiple spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is less than the downsampling multiple of the next spatial self-attention layer. Accordingly, step S1406 includes:

[0443] The sample splicing result is input into the first spatial self-attention layer in the spatial self-attention encoder; the output results of each spatial self-attention layer are fused to obtain the sample space feature vector.

[0444] In the embodiment of the present application, the process of obtaining the sample space feature vector can refer to the process of obtaining the to-be-detected space feature vector in the above step S1052, which will not be described in detail here.

[0445] In this way, the sample splicing results can be obtained corresponding to the encoding results under different downsampling multiples, that is, the generation of multi-scale feature maps with different resolutions can be achieved. Correspondingly, the fused sample space feature vector can also retain the characteristics of target objects of different scales (sizes), thus improving the detection ability of the target object detection model for target objects of different scales (sizes).

[0446] In some embodiments, each sample holographic data is obtained by the following steps:

[0447] S1: For each preset scanning angle, obtain echo information obtained by the radar scanning the sample space at the preset scanning angle as first sample echo information.

[0448] Among them, the first sample echo information under a preset scanning angle includes: the echo signal at each observation unit obtained by scanning under the preset scanning angle; the multiple preset scanning angles include: multiple preset front scanning angles and multiple preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans on one side of the sample space; the preset back scanning angle represents the scanning angle when scanning on the other side of the sample space.

[0449] S2: The first sample echo information obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle.

[0450] The difference between the specified front scanning angle and the specified back scanning angle is 180°.

[0451] S3: coherently synthesizing the second sample echo information at each preset front scanning angle to obtain front sample echo information, and coherently synthesizing the second sample echo information at each preset back scanning angle to obtain back sample echo information.

[0452] S4: Based on the preset observation time and time anti-convolution processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information.

[0453] The preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle.

[0454] S5: Process the omnidirectional sample echo information based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as sample holographic data.

[0455] In the embodiment of the present application, for step S1, the process of obtaining the first sample echo information at each preset scanning angle can refer to the process of obtaining the first echo information to be detected at each preset scanning angle in the above step S1101, which will not be repeated here.

[0456] For step S2, in order to ensure the uniqueness of the horizontal coordinate value, the vertical coordinate value, and the time used to characterize the same real position when the information of the same real position in the sample space is described later, the electronic device can adjust the first sample echo information at each preset scanning angle.

[0457] In one implementation, step S2 includes:

[0458] For the first sample echo information obtained at each preset scanning angle, the first sample echo information is adjusted according to the fourth formula to obtain the second sample echo information at each preset scanning angle.

[0459] Among them, the fourth formula is:

[0460] ;

[0461] Indicates a preset scanning angle, Indicates the observation unit in the first sample echo information under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second sample echo information under the preset scanning angle; Represents a collection; Represents a natural constant.

[0462] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle.

[0463] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle.

[0464] In the embodiment of the present application, the process of obtaining the second sample echo information at each preset scanning angle can refer to the process of obtaining the second echo information to be detected at each preset scanning angle in the above step S1102. The description of the fourth formula can refer to the relevant description of the above first formula, which is not repeated here.

[0465] Correspondingly, step S3 includes: based on the fifth formula, coherently synthesizing the second sample echo information at each preset front scanning angle to obtain the front sample echo information, and coherently synthesizing the second sample echo information at each preset back scanning angle to obtain the back sample echo information. The fifth formula is:

[0466] ;

[0467] ;

[0468] in, Indicates a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second sample echo information at the preset scanning angle at the preset scanning angle; Indicates the positive sample echo information, Indicates the back sample echo information; Indicates sum.

[0469] In the embodiment of the present application, the process of obtaining the front sample echo information can refer to the process of obtaining the front to-be-detected echo information in the above step S1103. The process of obtaining the back sample echo information can refer to the process of obtaining the back to-be-detected echo information in the above step S1103. The description of the fifth formula can refer to the relevant description of the second formula above, which will not be repeated here.

[0470] Correspondingly, step S4 includes: using the sixth formula, based on the preset observation time and time anti-folding processing, fusing the front sample echo information and the back sample echo information to obtain the omnidirectional sample echo information. The sixth formula is:

[0471] ;

[0472] Indicates one of the front sample echo information and the back sample echo information; represents the result obtained by processing the other sample echo information of the front sample echo information and the back sample echo information based on the preset observation time and time anti-convolution; T represents the preset observation time; t represents the time interval; Represents omni-directional sample echo information.

[0473] In the embodiment of the present application, the process of obtaining omnidirectional sample echo information can refer to the process of obtaining omnidirectional to-be-detected echo information in the above step S1104. The description of the sixth formula can refer to the relevant description of the above third formula, which will not be repeated here.

[0474] Based on the above processing, the echo signals obtained at each preset scanning angle can be fused to obtain complete three-dimensional holographic data for the sample space. Accordingly, the three-dimensional holographic data can fully represent the surface information of the scanned object in the sample space. Subsequently, target object detection can be performed based on the three-dimensional holographic data to ensure the accuracy of detection.

[0475] Moreover, during the above processing, the electronic device does not need to perform pre-processing such as cropping and filtering on the echo signal, which can also improve the accuracy of the three-dimensional holographic data and reduce noise interference. In this way, the information loss caused by pre-processing can be avoided, and the accuracy and resolution of target object detection can be further improved.

[0476] See also Fig.17 , Fig.17 An example diagram of the overall process of the target object detection model training process and prediction process provided in this application.

[0477] Fig.17In the process, the electronic device can obtain the echo information obtained by the radar scanning the space (sample space or space to be scanned) at each preset scanning angle, that is, multi-angle millimeter wave echo signal acquisition, and obtain multi-angle echo. Furthermore, the electronic device can fuse the obtained multiple echo information, that is, perform millimeter wave omnidirectional holographic data fusion.

[0478] In the training phase, the fused three-dimensional holographic data represents: sample holographic data. Accordingly, the electronic device can use the sample holographic data to perform training according to the above steps S1401 to S1409 to obtain a target object detection model. That is, using omnidirectional holographic data (training), the stage learning strategy based on the millimeter wave feature ambiguity is used for training to obtain a millimeter wave holographic large model based on the Transform architecture.

[0479] In the test phase, the fused three-dimensional holographic data represents: the holographic data to be detected. Accordingly, the electronic device can use the target detection model to process the holographic data to be detected according to the above steps S101 to S106 to obtain the detection result of the holographic data to be detected. That is, the omnidirectional holographic data (test) is input into the millimeter wave holographic large model based on the Transformer architecture (i.e., the target object detection model in this application) to obtain the (i.e., target object) detection result.

[0480] Based on the same inventive concept, the present application embodiment provides a target object detection device. Fig.18 , Fig.18 A structural diagram of a target object detection device provided in an embodiment of the present application, the device comprising:

[0481] The to-be-detected holographic data acquisition unit 1801 is used to acquire three-dimensional holographic data obtained by scanning the to-be-scanned space through a radar as the to-be-detected holographic data; wherein one three-dimensional holographic data includes multiple slices of the same size; one pixel point in one slice corresponds to a real position in the to-be-scanned space, the value of the pixel point includes characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to the pixels in one slice have the same scanning depth;

[0482] A first division unit 1802 is used to divide each slice of the to-be-detected holographic data into a plurality of to-be-detected spatial blocks in the same division manner;

[0483] The first grouping unit 1803 is used to select a plurality of to-be-detected spatial blocks having the same position in the corresponding slice from the to-be-detected spatial blocks obtained by the division, to obtain a to-be-detected spatial block group;

[0484] A first depth encoding unit 1804 is used to encode each to-be-detected spatial block group based on a deep self-attention mechanism to obtain a to-be-detected depth feature vector corresponding to each to-be-detected spatial block group;

[0485] A first spatial encoding unit 1805 is used to encode the splicing results of the depth feature vectors to be detected based on the spatial self-attention mechanism to obtain a spatial feature vector to be detected;

[0486] The detection result acquisition unit 1806 is used to perform detection using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

[0487] In some embodiments, the to-be-detected holographic data acquisition unit 1801 includes:

[0488] The first to-be-detected echo information acquisition subunit is used to acquire, for each preset scanning angle, echo information obtained by the radar when scanning the to-be-scanned space at the preset scanning angle as the first to-be-detected echo information; wherein the first to-be-detected echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the to-be-scanned space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the to-be-scanned space;

[0489] The second to-be-detected echo information acquisition subunit is used to adjust the first to-be-detected echo information obtained at each preset front scanning angle to the same specified front scanning angle, and obtain the second to-be-detected echo information at each preset front scanning angle; and adjust the first to-be-detected echo information obtained at each preset back scanning angle to the same specified back scanning angle, and obtain the second to-be-detected echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0490] The first coherent synthesis subunit is used to perform coherent synthesis on the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and to perform coherent synthesis on the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0491] The first fusion subunit is used to fuse the front echo information to be detected and the back echo information to be detected based on the preset observation time and time anti-convolution processing to obtain the omnidirectional echo information to be detected; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0492] The to-be-detected holographic data acquisition subunit is used to process the omnidirectional to-be-detected echo information based on the radar holographic imaging algorithm to obtain three-dimensional holographic data as the to-be-detected holographic data.

[0493] In some embodiments, the second to-be-detected echo information acquisition subunit is specifically used to:

[0494] For the first echo information to be detected obtained at each preset scanning angle, the first echo information to be detected is adjusted according to the first formula to obtain the second echo information to be detected at each preset scanning angle;

[0495] Among them, the first formula is:

[0496] ;

[0497] θ represents a preset scanning angle, Indicates the observation unit in the first echo information to be detected under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second echo information to be detected under the preset scanning angle; Represents a collection; represents a natural constant;

[0498] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0499] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0500] The first coherent synthesis subunit is specifically used for:

[0501] Based on the second formula, coherently synthesize the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and coherently synthesize the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected;

[0502] Wherein, the second formula is:

[0503] ;

[0504] ;

[0505] Where θ represents a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second echo information to be detected under the preset scanning angle; Indicates the positive echo information to be detected. Indicates the back echo information to be detected; Indicates sum.

[0506] In some embodiments, the first fusion subunit is specifically used for:

[0507] Using the third formula, based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain the omnidirectional echo information to be detected;

[0508] Wherein, the third formula is:

[0509] ;

[0510] Indicates one of the front echo information to be detected and the back echo information to be detected; represents the result obtained by processing the other of the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution; T represents the preset observation time; t represents time; Indicates the echo information to be detected in all directions.

[0511] In some embodiments, the first depth encoding unit 1804 is specifically configured to:

[0512] Input each to-be-detected spatial block group into the deep self-attention encoder in the pre-trained target object detection model to obtain the to-be-detected deep feature vector corresponding to each to-be-detected spatial block group; wherein the pre-trained target object detection model is: trained based on the sample holographic data and the label corresponding to each sample holographic data; the pre-trained target object detection model also includes: a spatial self-attention encoder;

[0513] The first spatial coding unit 1805 includes:

[0514] The detection splicing result acquisition subunit is used to splice the depth feature vectors to be detected according to the positions of the to-be-detected spatial blocks contained in the corresponding to-be-detected spatial block group in the corresponding slice to obtain the to-be-detected splicing result;

[0515] The first spatial encoding subunit is used to input the splicing result to be detected into the spatial self-attention encoder to obtain a spatial feature vector to be detected.

[0516] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0517] The first spatial encoding subunit is specifically used for:

[0518] Inputting the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder;

[0519] The output results of each spatial self-attention layer are fused to obtain the spatial feature vector to be detected.

[0520] In some embodiments, the pre-trained target object detection model further includes: a detection head;

[0521] The detection result acquisition unit 1806 is specifically used to:

[0522] Inputting the spatial feature vector to be detected into the detection head to obtain a prediction frame and a confidence level corresponding to each prediction frame; wherein the confidence level corresponding to a prediction frame indicates the probability that the prediction frame contains the target object;

[0523] If there is a prediction box whose confidence is greater than a preset confidence threshold, it is determined that the target object exists in the space to be scanned.

[0524] Based on the same inventive concept, the present application embodiment provides a model training device. Fig.19 , Fig.19 A structural diagram of a model training device provided in an embodiment of the present application, the device comprising:

[0525] The sample holographic data acquisition unit 1901 is used to acquire the sample holographic data of the current batch and the labels of each sample holographic data; wherein, the label of a sample holographic data represents the truth box in the sample holographic data, and each truth box represents the area occupied by a target object; a sample holographic data includes multiple slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, and the value of the pixel point includes the characteristic information of the echo signal obtained by scanning the real position, and the real positions corresponding to each pixel point in a slice have the same scanning depth;

[0526] The second division unit 1902 is used for dividing each slice of each sample holographic data of the current batch into a plurality of sample space blocks according to the same division method;

[0527] The second grouping unit 1903 is used to select a plurality of sample space blocks having the same position in the corresponding slice from the divided sample space blocks to obtain a sample space block group;

[0528] The second depth encoding unit 1904 is used to input each sample space block group into the depth self-attention encoder in the target object detection model in the initial state to obtain the sample depth feature vector corresponding to each sample space block group; wherein the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head;

[0529] The sample splicing result acquisition unit 1905 is used to splice the depth feature vectors of each sample according to the position of the sample space blocks included in the corresponding sample space block group in the corresponding slice to obtain the sample splicing result;

[0530] The second spatial encoding unit 1906 is used to input the sample splicing result into the spatial self-attention encoder to obtain a sample spatial feature vector;

[0531] The prediction result acquisition unit 1907 is used to input the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data;

[0532] The loss value calculation unit 1908 is used to calculate the loss value based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data;

[0533] The model parameter adjustment unit 1909 is used to adjust the model parameters of the target object detection model in the initial state based on the obtained loss value.

[0534] In some embodiments, the current batch of sample holographic data is selected from the full amount of sample holographic data; the target object detection model in the initial state is obtained by pre-training based on high-quality sample holographic data and labels corresponding to each high-quality sample holographic data; the image blurriness of the high-quality sample holographic data is lower than the image blurriness of other sample holographic data in the full amount of sample holographic data.

[0535] In some embodiments, the prediction result of a sample holographic data includes: a prediction box in the sample holographic data, and a confidence level corresponding to each prediction box; the confidence level corresponding to a prediction box indicates the probability that the prediction box contains the target object;

[0536] The loss value calculation unit 1908 includes:

[0537] The fuzziness calculation subunit is used to calculate the fuzziness of each truth frame in the label of the sample holographic data; wherein the fuzziness of a truth frame represents: the detection difficulty of the current target object detection model for the target object in the truth frame;

[0538] The loss value calculation subunit is used to calculate the loss value by using the position difference between a true value frame whose ambiguity is not greater than the current ambiguity threshold and a prediction frame corresponding to the true value frame, as well as the confidence corresponding to the prediction frame.

[0539] In some embodiments, the apparatus further comprises:

[0540] The fuzziness threshold adjustment unit is used to increase the current fuzziness threshold after adjusting the model parameters of the target object detection model in the initial state based on the loss value obtained from the sample holographic data of the current batch.

[0541] In some embodiments, the ambiguity calculation subunit is specifically used to:

[0542] For each true value frame in the label of the sample holographic data, respectively calculate the intersection-over-union ratio between the true value frame and each predicted frame;

[0543] Calculate the product of each intersection-over-union ratio and the confidence of the prediction box used to obtain the intersection-over-union ratio;

[0544] The inverse of the maximum value in each product is determined as the fuzziness of the true value frame.

[0545] In some embodiments, the spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer;

[0546] The second spatial coding unit 1906 is specifically configured to:

[0547] Inputting the sample splicing result into the first spatial self-attention layer in the spatial self-attention encoder;

[0548] The output results of each spatial self-attention layer are fused to obtain the sample space feature vector.

[0549] In some embodiments, each sample holographic data is obtained by the following steps:

[0550] For each preset scanning angle, echo information obtained by the radar scanning the sample space at the preset scanning angle is obtained as first sample echo information; wherein the first sample echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the sample space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the sample space;

[0551] The first sample echo information obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°;

[0552] Performing coherent synthesis on the second sample echo information at each preset front scanning angle to obtain front sample echo information, and performing coherent synthesis on the second sample echo information at each preset back scanning angle to obtain back sample echo information;

[0553] Based on the preset observation time and time anti-convolution processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle;

[0554] The omnidirectional sample echo information is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as sample holographic data.

[0555] In some embodiments, the first sample echo information obtained at each preset front scanning angle is adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle, including:

[0556] For the first sample echo information obtained at each preset scanning angle, the first sample echo information is adjusted according to the fourth formula to obtain the second sample echo information at each preset scanning angle;

[0557] Wherein, the fourth formula is:

[0558] ;

[0559] Indicates a preset scanning angle, Indicates the observation unit in the first sample echo information under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second sample echo information under the preset scanning angle; Represents a collection; represents a natural constant;

[0560] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively;

[0561] The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively;

[0562] The coherent synthesis of the second sample echo information at each preset front scanning angle to obtain the front sample echo information, and the coherent synthesis of the second sample echo information at each preset back scanning angle to obtain the back sample echo information, include:

[0563] Based on the fifth formula, the second sample echo information at each preset front scanning angle is coherently synthesized to obtain the front sample echo information, and the second sample echo information at each preset back scanning angle is coherently synthesized to obtain the back sample echo information;

[0564] Wherein, the fifth formula is:

[0565] ;

[0566] ;

[0567] in, Indicates a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second sample echo information at the preset scanning angle at the preset scanning angle; Indicates the positive sample echo information, Indicates the back sample echo information; Indicates sum.

[0568] In some embodiments, the front sample echo information and the back sample echo information are fused based on the preset observation time and time anti-convolution processing to obtain omnidirectional sample echo information, including:

[0569] Using the sixth formula, based on the preset observation time and time anti-folding processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information;

[0570] Wherein, the sixth formula is:

[0571] ;

[0572] Indicates one of the front sample echo information and the back sample echo information; represents the result obtained by processing the other sample echo information of the front sample echo information and the back sample echo information based on the preset observation time and time anti-convolution; T represents the preset observation time; t represents the time interval; Represents omni-directional sample echo information.

[0573] In the technical solution of the present application, the operations involved in acquiring, storing, using, processing, transmitting, providing and disclosing three-dimensional holographic data are all carried out with the authorization of the user.

[0574] The present application also provides an electronic device, such as Fig. 20 As shown, including:

[0575] Memory 2001, used for storing computer programs;

[0576] The processor 2002 is used to implement any of the above-mentioned target object detection methods or steps of the model training method when executing the program stored in the memory 2001.

[0577] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 2002, the communication interface, and the memory 2001 communicate with each other via the communication bus.

[0578] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0579] The communication interface is used for communication between the above electronic device and other devices.

[0580] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0581] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0582] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned target object detection methods or model training methods are implemented.

[0583] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any target object detection method or model training method in the above-mentioned embodiments.

[0584] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0585] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0586] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0587] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target object detection method, characterized in that: The method comprises: Acquire three-dimensional holographic data obtained by scanning the space to be scanned by radar as the holographic data to be detected; wherein one three-dimensional holographic data includes multiple slices of the same size; one pixel point in one slice corresponds to a real position in the space to be scanned, the value of the pixel point includes characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to the pixels in one slice have the same scanning depth; According to the same division method, each slice of the to-be-detected holographic data is divided into a plurality of to-be-detected spatial blocks; Selecting multiple to-be-detected space blocks having the same position in the corresponding slices from the to-be-detected space blocks obtained by the division to obtain a to-be-detected space block group; Based on the deep self-attention mechanism, each space block group to be detected is encoded respectively to obtain the depth feature vector to be detected corresponding to each space block group to be detected; Based on the spatial self-attention mechanism, the concatenation results of the depth feature vectors to be detected are encoded to obtain the spatial feature vector to be detected; Detection is performed using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

2. The method according to claim 1, characterized in that The step of acquiring three-dimensional holographic data obtained by scanning the space to be scanned by the radar as the holographic data to be detected comprises: For each preset scanning angle, the echo information obtained by the radar when scanning the space to be scanned at the preset scanning angle is obtained as the first echo information to be detected; wherein the first echo information to be detected at a preset scanning angle includes: the echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: multiple preset front scanning angles and multiple preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the space to be scanned; the preset back scanning angle represents the scanning angle when scanning the other side of the space to be scanned; The first echo information to be detected obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second echo information to be detected at each preset front scanning angle; and the first echo information to be detected obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second echo information to be detected at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°; Performing coherent synthesis on the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and performing coherent synthesis on the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected; Based on the preset observation time and time anti-convolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain all-round echo information to be detected; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle; The omnidirectional echo information to be detected is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as the holographic data to be detected.

3. The method according to claim 2, characterized in that The first echo information to be detected obtained at each preset frontal scanning angle is respectively adjusted to the same specified frontal scanning angle to obtain the second echo information to be detected at each preset frontal scanning angle; and adjusting the first echo information to be detected obtained at each preset back scanning angle to the same specified back scanning angle to obtain the second echo information to be detected at each preset back scanning angle, including: For the first echo information to be detected obtained at each preset scanning angle, the first echo information to be detected is adjusted according to the first formula to obtain the second echo information to be detected at each preset scanning angle; Among them, the first formula is: ； θ represents a preset scanning angle, Indicates the observation unit in the first echo information to be detected under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second echo information to be detected under the preset scanning angle; Represents a collection; represents a natural constant; The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively; The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively; The coherent synthesis of the second echo information to be detected at each preset front scanning angle to obtain the front echo information to be detected, and the coherent synthesis of the second echo information to be detected at each preset back scanning angle to obtain the back echo information to be detected, include: Based on the second formula, the second echo information to be detected at each preset front scanning angle is coherently synthesized to obtain the front echo information to be detected, and the second echo information to be detected at each preset back scanning angle is coherently synthesized to obtain the back echo information to be detected; Wherein, the second formula is: ； ； Where θ represents a preset scanning angle, Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second echo information to be detected under the preset scanning angle; Indicates the positive echo information to be detected. Indicates the back echo information to be detected; Indicates sum.

4. The method according to claim 3, characterized in that Based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain the omnidirectional echo information to be detected, including: Using the third formula, based on the preset observation time and time deconvolution processing, the front echo information to be detected and the back echo information to be detected are fused to obtain the omnidirectional echo information to be detected; Wherein, the third formula is: ； Indicates one of the front echo information to be detected and the back echo information to be detected; represents the result obtained by processing the other of the front echo information to be detected and the back echo information to be detected based on the preset observation time and time deconvolution; T represents the preset observation time; t represents time; Indicates the echo information to be detected in all directions.

5. The method according to any one of claims 1 to 4, characterized in that: The method of encoding each to-be-detected spatial block group based on the deep self-attention mechanism to obtain a to-be-detected depth feature vector corresponding to each to-be-detected spatial block group includes: Input each to-be-detected spatial block group into the deep self-attention encoder in the pre-trained target object detection model to obtain the to-be-detected deep feature vector corresponding to each to-be-detected spatial block group; wherein the pre-trained target object detection model is: trained based on the sample holographic data and the label corresponding to each sample holographic data; the pre-trained target object detection model also includes: a spatial self-attention encoder; The splicing result of each depth feature vector to be detected is encoded based on the spatial self-attention mechanism to obtain the spatial feature vector to be detected, including: According to the position of the to-be-detected spatial slices contained in the corresponding to-be-detected spatial slice group in the corresponding slice, each to-be-detected depth feature vector is spliced ​​to obtain a to-be-detected splicing result; The splicing result to be detected is input into the spatial self-attention encoder to obtain a spatial feature vector to be detected.

6. The method according to claim 5, characterized in that The spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer; The splicing result to be detected is input into the spatial self-attention encoder to obtain the spatial feature vector to be detected, including: Inputting the splicing result to be detected into the first spatial self-attention layer in the spatial self-attention encoder; The output results of each spatial self-attention layer are fused to obtain the spatial feature vector to be detected.

7. The method according to claim 5, characterized in that The pre-trained object detection model also includes: a detection head; The detecting by using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned includes: Inputting the spatial feature vector to be detected into the detection head to obtain a prediction frame and a confidence level corresponding to each prediction frame; wherein the confidence level corresponding to a prediction frame indicates the probability that the prediction frame contains the target object; If there is a prediction box whose confidence is greater than a preset confidence threshold, it is determined that the target object exists in the space to be scanned.

8. A model training method, characterized in that: The method comprises: Obtain the sample holographic data of the current batch and the labels of each sample holographic data; wherein the label of a sample holographic data represents the true value frame in the sample holographic data, and each true value frame represents the area occupied by a target object; a sample holographic data comprises multiple slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, the value of the pixel point comprises the characteristic information of the echo signal obtained by scanning the real position, and the real positions corresponding to each pixel point in a slice have the same scanning depth; For each sample holographic data of the current batch, each slice in the sample holographic data is divided into a plurality of sample space blocks according to the same division method; Selecting multiple sample space blocks having the same position in the corresponding slice from the sample space blocks obtained by the division to obtain a sample space block group; Input each sample space block group into the deep self-attention encoder in the target object detection model in the initial state to obtain the sample depth feature vector corresponding to each sample space block group; wherein the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head; According to the position of the sample space blocks included in the corresponding sample space block group in the corresponding slice, the depth feature vectors of each sample are spliced ​​to obtain a sample splicing result; Input the sample splicing result into the spatial self-attention encoder to obtain a sample space feature vector; Inputting the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data; Calculate the loss value based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data; Based on the obtained loss value, the model parameters of the target object detection model in the initial state are adjusted.

9. The method according to claim 8, characterized in that The current batch of sample holographic data is selected from the full amount of sample holographic data; the target object detection model in the initial state is obtained by pre-training based on high-quality sample holographic data and labels corresponding to each high-quality sample holographic data; the image blurriness of the high-quality sample holographic data is lower than the image blurriness of other sample holographic data in the full amount of sample holographic data.

10. The method according to claim 8, characterized in that The prediction result of a sample holographic data includes: a prediction frame in the sample holographic data, and a confidence level corresponding to each prediction frame; the confidence level corresponding to a prediction frame indicates the probability that the prediction frame contains the target object; The calculation of the loss value based on the difference between the prediction result of the sample holographic data and the label corresponding to the sample holographic data includes: For each truth frame in the label of the sample holographic data, the fuzziness of the truth frame is calculated; wherein the fuzziness of a truth frame represents: the detection difficulty of the current target object detection model for the target object in the truth frame; The loss value is calculated using the position difference between a true value box whose ambiguity is not greater than the current ambiguity threshold and a prediction box corresponding to the true value box, as well as the confidence corresponding to the prediction box.

11. The method according to claim 10, characterized in that The method further comprises: After adjusting the model parameters of the target object detection model in the initial state based on the loss value obtained from the current batch of sample holographic data, the current blur threshold is increased.

12. The method according to claim 10, characterized in that The step of calculating the fuzziness of each truth frame in the label of the sample holographic data comprises: For each true value frame in the label of the sample holographic data, respectively calculate the intersection-over-union ratio between the true value frame and each predicted frame; Calculate the product of each intersection-over-union ratio and the confidence of the prediction box used to obtain the intersection-over-union ratio; The inverse of the maximum value in each product is determined as the fuzziness of the true value frame.

13. The method according to claim 8, characterized in that The spatial self-attention encoder comprises a plurality of spatial self-attention layers connected in series and having different downsampling multiples; in two adjacent spatial self-attention layers, the downsampling multiple of the previous spatial self-attention layer is smaller than the downsampling multiple of the next spatial self-attention layer; The sample concatenation result is input into the spatial self-attention encoder to obtain a sample space feature vector, including: Inputting the sample splicing result into the first spatial self-attention layer in the spatial self-attention encoder; The output results of each spatial self-attention layer are fused to obtain the sample space feature vector.

14. The method according to claim 8, characterized in that Each sample holographic data is obtained through the following steps: For each preset scanning angle, echo information obtained by the radar scanning the sample space at the preset scanning angle is obtained as first sample echo information; wherein the first sample echo information at a preset scanning angle includes: an echo signal at each observation unit obtained by scanning at the preset scanning angle; the multiple preset scanning angles include: a plurality of preset front scanning angles and a plurality of preset back scanning angles; the preset front scanning angle represents the scanning angle when the radar scans one side of the sample space; the preset back scanning angle represents the scanning angle when the radar scans the other side of the sample space; The first sample echo information obtained at each preset front scanning angle is respectively adjusted to the same specified front scanning angle to obtain the second sample echo information at each preset front scanning angle; and the first sample echo information obtained at each preset back scanning angle is respectively adjusted to the same specified back scanning angle to obtain the second sample echo information at each preset back scanning angle; wherein the specified front scanning angle differs from the specified back scanning angle by 180°; Performing coherent synthesis on the second sample echo information at each preset front scanning angle to obtain front sample echo information, and performing coherent synthesis on the second sample echo information at each preset back scanning angle to obtain back sample echo information; Based on the preset observation time and time anti-convolution processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information; wherein the preset observation time represents: for the same real position, the sum of the observation time of scanning the position according to the specified front scanning angle and the observation time of scanning the position according to the specified back scanning angle; The omnidirectional sample echo information is processed based on a radar holographic imaging algorithm to obtain three-dimensional holographic data as sample holographic data.

15. The method according to claim 14, characterized in that The first sample echo information obtained at each preset frontal scanning angle is respectively adjusted to the same specified frontal scanning angle to obtain the second sample echo information at each preset frontal scanning angle; and adjusting the first sample echo information obtained at each preset back side scanning angle to the same specified back side scanning angle to obtain the second sample echo information at each preset back side scanning angle, including: For the first sample echo information obtained at each preset scanning angle, the first sample echo information is adjusted according to the fourth formula to obtain the second sample echo information at each preset scanning angle; Wherein, the fourth formula is: ； Indicates a preset scanning angle. Indicates the observation unit in the first sample echo information under the preset scanning angle The echo signal at , t represents the time, represents the i-th horizontal and k-th vertical observation unit under the preset scanning angle, Indicates the observation unit at the preset scanning angle The lateral position compensation, Indicates the observation unit at the preset scanning angle The longitudinal position compensation, Indicates the observation unit at the preset scanning angle The amplitude compensation amount, Indicates the observation unit at the preset scanning angle The phase compensation amount; Indicates the second sample echo information under the preset scanning angle; Represents a collection; represents a natural constant; The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset frontal scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset frontal scanning angle and the specified frontal scanning angle respectively; The lateral position compensation, longitudinal position compensation, amplitude compensation, and phase compensation of each observation unit at each preset back scanning angle are all determined by the difference between the echo signals obtained by scanning the standard part at the preset back scanning angle and the specified back scanning angle respectively; The coherent synthesis of the second sample echo information at each preset front scanning angle to obtain the front sample echo information, and the coherent synthesis of the second sample echo information at each preset back scanning angle to obtain the back sample echo information, include: Based on the fifth formula, the second sample echo information at each preset front scanning angle is coherently synthesized to obtain the front sample echo information, and the second sample echo information at each preset back scanning angle is coherently synthesized to obtain the back sample echo information; Wherein, the fifth formula is: ； ； in, Indicates a preset scanning angle. Indicates that the preset scanning angle is the preset front scanning angle. Indicates that the preset scanning angle is the preset back scanning angle; Indicates the second sample echo information at the preset scanning angle at the preset scanning angle; Indicates the positive sample echo information, Indicates the back sample echo information; Indicates sum.

16. The method according to claim 15, characterized in that The front sample echo information and the back sample echo information are fused based on the preset observation time and time anti-folding processing to obtain omnidirectional sample echo information, including: Using the sixth formula, based on the preset observation time and time anti-folding processing, the front sample echo information and the back sample echo information are fused to obtain omnidirectional sample echo information; Wherein, the sixth formula is: ； Indicates one of the front sample echo information and the back sample echo information; represents the result obtained by processing the other sample echo information of the front sample echo information and the back sample echo information based on the preset observation time and time anti-convolution; T represents the preset observation time; t represents the time interval; Represents omni-directional sample echo information.

17. A target object detection device, characterized in that: The device comprises: The holographic data acquisition unit to be detected is used to acquire three-dimensional holographic data obtained by scanning the space to be scanned by radar as the holographic data to be detected; wherein one three-dimensional holographic data includes multiple slices of the same size; one pixel point in one slice corresponds to a real position in the space to be scanned, the value of the pixel point includes characteristic information of an echo signal obtained by scanning the real position, and the real positions corresponding to the pixels in one slice have the same scanning depth; A first division unit, used for dividing each slice of the to-be-detected holographic data into a plurality of to-be-detected spatial blocks according to the same division method; A first grouping unit is used to select a plurality of to-be-detected spatial blocks having the same position in the corresponding slice from the to-be-detected spatial blocks obtained by division, to obtain a to-be-detected spatial block group; A first depth encoding unit is used to encode each to-be-detected spatial block group based on a deep self-attention mechanism to obtain a to-be-detected depth feature vector corresponding to each to-be-detected spatial block group; A first spatial encoding unit is used to encode the splicing results of the obtained depth feature vectors to be detected based on a spatial self-attention mechanism to obtain a spatial feature vector to be detected; The detection result acquisition unit is used to perform detection using the feature vector of the space to be detected to obtain a detection result representing the target object in the space to be scanned.

18. A model training device, characterized in that: The device comprises: The sample holographic data acquisition unit is used to acquire the sample holographic data of the current batch and the labels of each sample holographic data; wherein the label of a sample holographic data represents the truth frame in the sample holographic data, and each truth frame represents the area occupied by a target object; a sample holographic data comprises a plurality of slices of the same size; a pixel point in a slice corresponds to a real position in the sample space, the value of the pixel point comprises the characteristic information of the echo signal obtained by scanning the real position, and the scanning depth of the real position corresponding to each pixel point in a slice is the same; A second division unit is used for dividing each slice of each sample holographic data of the current batch into a plurality of sample space blocks according to the same division method; A second grouping unit is used to select a plurality of sample space blocks having the same position in the corresponding slice from the sample space blocks obtained by the division, so as to obtain a sample space block group; A second depth encoding unit is used to input each sample space block group into a deep self-attention encoder in the target object detection model in the initial state to obtain a sample depth feature vector corresponding to each sample space block group; wherein the target object detection model in the initial state also includes: a spatial self-attention encoder and a detection head; A sample splicing result acquisition unit is used to splice the depth feature vectors of each sample according to the position of the sample space blocks included in the corresponding sample space block group in the corresponding slice to obtain a sample splicing result; A second spatial encoding unit, used for inputting the sample splicing result into the spatial self-attention encoder to obtain a sample spatial feature vector; A prediction result acquisition unit, used for inputting the sample space feature vector into the detection head to obtain the prediction result of the sample holographic data; a loss value calculation unit, configured to calculate a loss value based on a difference between a prediction result of the sample holographic data and a label corresponding to the sample holographic data; The model parameter adjustment unit is used to adjust the model parameters of the target object detection model in the initial state based on the obtained loss value.

19. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7, or any of the methods described in claims 8-16, when executing a program stored in a memory.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1-7, or any one of claims 8-16 is implemented.

21. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7 or any one of claims 8 to 16.

Images