Image target fusion detection processing method and device

Abstract

The invention discloses an image target fusion detection processing method and device. The method comprises the following steps: acquiring a target fusion training data set; the target fusion training data set comprises an image data subset, an ultrasonic data subset and a corresponding label subset; the ultrasonic data subset comprises ultrasonic data; the image data subset comprises image data; preprocessing the target fusion training data set to obtain a preprocessed training data set; performing training processing on the constructed fusion detection model by using the pre-processed training data set to obtain a trained fusion detection model; and processing the acquired image data and ultrasonic data by using the trained fusion detection model to obtain target information.

Description

Technical Field

[0001] The present invention relates to the fields of communication and artificial intelligence, and in particular to a method and device for fusion detection and processing of image targets. Background Art

[0002] With the widespread application of unmanned motion platforms and unmanned driving technologies, perception and obstacle avoidance are the core issues that need to be addressed, and image target detection during motion has become one of the key technologies for perception and obstacle avoidance. Efficient and stable target detection can ensure the route planning and obstacle avoidance control of unmanned vehicles, and is the basis for realizing autonomous driving of unmanned vehicles. In view of the working characteristics of unmanned vehicles, such as low altitude and large changes in ambient light, traditional target detection methods have certain limitations and it is difficult to achieve real-time and accurate results.

[0003] How to achieve efficient and accurate target detection in images collected in unmanned working scenarios is a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention mainly solves the problem of how to achieve efficient and accurate target detection for images collected in unmanned vehicle working scenarios. The present invention discloses a method and device for fusion detection processing of image targets.

[0005] In a first aspect of the embodiments of the present application, a method for fusion detection and processing of an image target is disclosed, comprising:

[0006] S1, obtaining a target fusion training data set; the target fusion training data set includes an image data subset, an ultrasonic data subset and a corresponding label subset; the ultrasonic data subset includes ultrasonic data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasonic data in the image data subset and the ultrasonic data subset, respectively;

[0007] S2, preprocessing the target fusion training data set to obtain a preprocessed training data set;

[0008] S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model;

[0009] S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information.

[0010] The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes:

[0011] S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set;

[0012] S22, performing boundary check processing on the first data set to obtain a preprocessed training data set.

[0013] The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes:

[0014] S221, for each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value of the value range and a lower limit value of the value range;

[0015] S222, for each data of each data attribute of the first data set, judging whether the value of the data is within the value range according to the value range of the data attribute; if the data is within the value range, not processing the data; if the data is not within the value range, setting the value of the data to the upper limit value or the lower limit value of the value range that is closest to the data;

[0016] S223, after completing S222 for the data of all data attributes, a preprocessed training data set is obtained.

[0017] The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes:

[0018] S2201, for each type of data attribute of the first data set, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first data set; if it is not greater than the first regression discrimination threshold, do not process the data;

[0019] S2202, after completing S2201 for all data attributes, a preprocessed training data set is obtained.

[0020] The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module;

[0021] The feature encoding module includes an input submodule, a first feature encoding submodule and a second feature encoding submodule; the input submodule is connected to the first feature encoding submodule and the second feature encoding submodule;

[0022] The input submodule includes an input port, a first output port and a second output port, and is used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port;

[0023] The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0024] The first fusion output module is used to output a first fusion signal;

[0025] The second fusion output module is used to output a second fusion signal;

[0026] The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the first feature coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0027] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the first feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the first feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the first feature coding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0028] The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0029] The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0030] The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the second feature encoding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0031] The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the second feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the second feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the second feature coding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module;

[0032] The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0033] The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module;

[0034] The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module;

[0035] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module;

[0036] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module;

[0037] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module;

[0038] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module;

[0039] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0040] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0041] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0042] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively;

[0043] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0044] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0045] The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0046] The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module;

[0047] The first PReLU activation function module is used to input a first fusion signal;

[0048] The fifth PReLU activation function module is used to input the second fusion signal;

[0049] The ninth fully connected layer is used to input image data;

[0050] The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input into the fusion detection model.

[0051] The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes:

[0052] S301, initializing a first training number value;

[0053] S302, acquiring ultrasonic data from the preprocessed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307;

[0054] S303, inputting the image data in the preprocessed training data set into the fusion detection model; determining the label data corresponding to the ultrasonic data and the image data input into the fusion detection model as label information; in each execution of S303, the image data input into the fusion detection model is different;

[0055] S304, using the fusion detection model to process the input information to obtain predicted image target information;

[0056] S305, calculating the predicted image target information and label information using a loss function to obtain a first difference value;

[0057] S306, performing an accumulation operation on the first training times;

[0058] S307, determining whether the first training times value exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308;

[0059] When the first judgment result is no, determining whether the first difference value is less than a set convergence threshold, and obtaining a second judgment result;

[0060] When the second judgment result is yes, execute S308; when the second judgment result is no, update the parameters of the feature encoding module and the fusion detection module using the update model, and execute S303;

[0061] S308, initializing the second training times value;

[0062] S309, acquiring image data from the preprocessed training data set, and inputting the acquired image data into a fusion detection model; the acquired image data remains unchanged in S310 to S314;

[0063] S310, inputting the ultrasonic data in the preprocessed training data set into the feature coding module in the fusion detection model; determining the label data corresponding to the image data and ultrasonic data input into the fusion detection model as label information; in each execution of S310, the ultrasonic data input into the feature coding module is different;

[0064] S311, using the fusion detection model to process the input information to obtain predicted image target information;

[0065] S312, calculating the predicted image target information and label information using a loss function to obtain a second difference value;

[0066] S313, performing an accumulation operation on the second training times value;

[0067] S314, determining whether the second training times value exceeds a preset second training times threshold value, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model;

[0068] When the third judgment result is no, determining whether the second difference value is less than a set convergence threshold, and obtaining a fourth judgment result;

[0069] When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

[0070] The update model is:

[0071]

[0072] θ←θ+v,

[0073] In the formula, is the first difference value or the second difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, It represents the partial derivative of the variable θ; the i-th input information includes ultrasonic data and image data.

[0074] According to a second aspect of an embodiment of the present invention, a fusion detection and processing device for an image target is disclosed, the device comprising:

[0075] A memory storing executable program code;

[0076] a processor coupled to the memory;

[0077] The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method.

[0078] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the image target fusion detection processing method.

[0079] According to a fourth aspect of an embodiment of the present invention, an information data processing terminal is disclosed. The information data processing terminal is used to implement the image target fusion detection processing method.

[0080] The beneficial effects of the present invention are:

[0081] The method of the present invention can improve the accuracy of target estimation, reduce the number of iterative calculations, improve calculation efficiency, and can also handle the deformation of moving targets in the field of view, thereby enhancing the adaptability to moving objects.

[0082] In terms of unmanned vehicle target detection, the present invention has brought about a relatively obvious effect. The present invention can deduce local details from the overall motion trend, enhance the robustness of processing image deformation, and effectively detect moving targets with drastic size changes under the perspective of unmanned vehicles. By establishing a neural network model, it can handle the lighting change environment and is no longer limited by lighting conditions, thus expanding the scope of application. The motion information is fed back to the control system, which can achieve effective avoidance of motion obstacles and enhance the autonomous obstacle avoidance capability of unmanned vehicles.

[0083] Experimental results show that the present invention retains the strengths of traditional methods in dealing with occlusion, small targets, etc., and through technological innovation enables it to handle more complex actual environments, including difficult scenes such as target deformation and lighting changes, achieving higher robustness, accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION

[0085] In order to better understand the content of the present invention, an embodiment is given here.

[0086] Figure 1 It is a flow chart for implementing the method of the present invention.

[0087] Aiming at the problem of realizing efficient and accurate target detection on images collected in unmanned working scenes, the present invention discloses a method and device for image target fusion detection processing.

[0088] In a first aspect of the embodiments of the present application, a method for fusion detection and processing of an image target is disclosed, comprising:

[0089] S1, obtaining a target fusion training data set; the target fusion training data set includes an image data subset, an ultrasonic data subset and a corresponding label subset; the ultrasonic data subset includes ultrasonic data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasonic data in the image data subset and the ultrasonic data subset, respectively;

[0090] S2, preprocessing the target fusion training data set to obtain a preprocessed training data set;

[0091] S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model;

[0092] S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information;

[0093] The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes:

[0094] S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set;

[0095] S22, performing boundary check processing on the first data set to obtain a preprocessed training data set.

[0096] The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes:

[0097] S221, for each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value of the value range and a lower limit value of the value range;

[0098] S222, for each data of each data attribute of the first data set, judging whether the value of the data is within the value range according to the value range of the data attribute; if the data is within the value range, not processing the data; if the data is not within the value range, setting the value of the data to the upper limit value or the lower limit value of the value range that is closest to the data;

[0099] S223, after completing S222 for the data of all data attributes, a preprocessed training data set is obtained.

[0100] The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes:

[0101] S2201, for each type of data attribute of the first data set, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first data set; if it is not greater than the first regression discrimination threshold, do not process the data;

[0102] S2202, after completing S2201 for all data attributes, a preprocessed training data set is obtained.

[0103] The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module;

[0104] The feature coding module includes an input submodule, a first feature coding submodule and a second feature coding submodule; the input submodule is connected to the first feature coding submodule and the second feature coding submodule; the input submodule includes an input port, a first output port and a second output port, which are used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port;

[0105] The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0106] The first fusion output module is used to output a first fusion signal;

[0107] The second fusion output module is used to output a second fusion signal;

[0108] The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the first feature coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0109] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the first feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the first feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the first feature coding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0110] The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0111] The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0112] The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the second feature encoding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0113] The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the second feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the second feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the second feature coding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module;

[0114] The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0115] The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module;

[0116] The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module;

[0117] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module;

[0118] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module;

[0119] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module;

[0120] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module;

[0121] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0122] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0123] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0124] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively;

[0125] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0126] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0127] The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0128] The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module;

[0129] The first PReLU activation function module is used to input a first fusion signal;

[0130] The fifth PReLU activation function module is used to input the second fusion signal;

[0131] The ninth fully connected layer is used to input image data;

[0132] The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input into the fusion detection model.

[0133] The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes:

[0134] S301, initializing a first training number value;

[0135] S302, acquiring ultrasonic data from the preprocessed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307;

[0136] S303, inputting the image data in the preprocessed training data set into the fusion detection model; determining the image target information corresponding to the ultrasonic data and the image data input into the fusion detection model as label information; in each execution of S303, the image data input into the fusion detection model is different;

[0137] S304, using the fusion detection model to process the input information to obtain predicted image target information;

[0138] S305, calculating the predicted image target information and label information using a loss function to obtain a difference value;

[0139] S306, performing an accumulation operation on the first training times;

[0140] S307, determining whether the first training times value exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308;

[0141] When the first judgment result is no, judging whether the difference value is less than a set convergence threshold, and obtaining a second judgment result;

[0142] When the second judgment result is yes, execute S308; when the second judgment result is no, update the parameters of the feature encoding module and the fusion detection module using the update model, and execute S303;

[0143] S308, initializing the second training times value;

[0144] S309, acquiring image data from the preprocessed training data set, and inputting the acquired image data into a fusion detection model; the acquired image data remains unchanged in S310 to S314;

[0145] S310, inputting the ultrasonic data in the preprocessed training data set into the feature coding module in the fusion detection model; determining the image target information corresponding to the image data and ultrasonic data input into the fusion detection model as label information; in each execution of S310, the ultrasonic data input into the feature coding module is different;

[0146] S311, using the fusion detection model to process the input information to obtain predicted image target information;

[0147] S312, using a loss function to calculate the predicted image target information and label information to obtain a difference value;

[0148] S313, performing an accumulation operation on the second training times value;

[0149] S314, determining whether the second training times value exceeds a preset second training times threshold value, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model;

[0150] When the third judgment result is no, judging whether the difference value is less than a set convergence threshold, and obtaining a fourth judgment result;

[0151] When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

[0152] The update model is:

[0153]

[0154] θ←θ+v,

[0155] In the formula, is the difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, It represents the partial derivative of the variable θ. The i-th input information includes ultrasonic data and image data.

[0156] The loss function may adopt a cross entropy loss function.

[0157] Whether the difference value satisfies the convergence condition refers to whether the difference value is less than a preset error threshold value; when the difference value is less than the preset error threshold value, it is judged that the difference value satisfies the convergence condition; when the difference value is greater than the preset error threshold value, it is judged that the difference value does not satisfy the convergence condition.

[0158] The autoregressive-moving average modeling can be implemented using the ARMA method;

[0159] The first feature encoding submodule is implemented using the ShuffleNetV2 network;

[0160] The second feature encoding submodule is implemented by using a one-dimensional convolutional neural network;

[0161] The outlier elimination includes filling missing values, smoothing noise data, smoothing or deleting outlier points; the smoothed noise data is first determined to obtain noise data, and then the noise data is smoothed according to the data before and after the noise data; the noise data is a value whose value is less than the detection sensitivity of the sensor of the observed data, or greater than the measurement upper limit of the sensor of the observed data. The outlier point can be determined by the Kalman filtering method. The filling value of the missing value can be determined by averaging the measured values ​​within a certain sampling interval before and after the missing value.

[0162] The autoregressive-moving average modeling can be achieved by performing curve fitting on the curve to be approximated using a function approximation method.

[0163] The curve fitting of the curve to be approximated by the function approximation method may be performed by using the best consistent linear approximation method. The best consistent approximation polynomial f(Ix) is expressed as:

[0164] f(Ix)=α P1 (Ix) P1 +α P1-1 (Ix) P1-1 +…+α 2 (Ix) 2 +α 1 (Ix)+α 0 ,

[0165] Where P1 is the order of the best consistent approximation polynomial f(Ix), α0, α1, α2, …, α P1 are the coefficients of the optimal consistent approximation polynomial f(Ix);

[0166] The data collection information includes time information, space information or capability type information corresponding to the data collection; when performing data fitting, any of the above information can be used as an independent variable, and the corresponding data value can be used as a dependent variable.

[0167] The first feature encoding submodule is implemented using the ShuffleNetV2 network;

[0168] The second feature encoding submodule is implemented by using a one-dimensional convolutional neural network;

[0169] The trained fusion detection model is used to process the collected image data and ultrasonic data to obtain target information, including:

[0170] Using the trained fusion detection model, the collected image data and ultrasonic data are processed to obtain a target occurrence probability value;

[0171] Determine whether the target occurrence probability value is greater than a preset probability threshold value. If so, set the target information as containing the target; if not, set the target information as not containing the target.

[0172] The target information is used to indicate whether the acquired image data and ultrasonic data contain a target;

[0173] The ultrasonic data is ultrasonic training data constructed by collecting data using an ultrasonic sensor installed on an unmanned motion platform;

[0174] The image data is image training data constructed by collecting data using an image sensor installed on an unmanned motion platform;

[0175] According to a second aspect of an embodiment of the present invention, a fusion detection and processing device for an image target is disclosed, the device comprising:

[0176] A memory storing executable program code;

[0177] a processor coupled to the memory;

[0178] The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method.

[0179] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the image target fusion detection processing method.

[0180] According to a fourth aspect of an embodiment of the present invention, an information data processing terminal is disclosed. The information data processing terminal is used to implement the image target fusion detection processing method.

[0181] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A fusion detection processing method for image targets, characterized in that: include: S1, obtain the target fusion training data set; The target fusion training data set includes an image data subset, an ultrasound data subset and a corresponding label subset; The ultrasonic data subset includes ultrasonic data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasonic data in the image data subset and the ultrasonic data subset, respectively; S2, preprocessing the target fusion training data set to obtain a preprocessed training data set; S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model; S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information.

2. The method for fusion detection of image targets according to claim 1, characterized in that: The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes: S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set; S22, performing boundary check processing on the first data set to obtain a preprocessed training data set.

3. The image target fusion detection processing method according to claim 1, characterized in that: The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes: S221, for each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value of the value range and a lower limit value of the value range; S222, for each data of each data attribute of the first data set, judging whether the value of the data is within the value range according to the value range of the data attribute; if the data is within the value range, not processing the data; if the data is not within the value range, setting the value of the data to the upper limit value or the lower limit value of the value range that is closest to the data; S223, after completing S222 for the data of all data attributes, a preprocessed training data set is obtained.

4. The method for fusion detection of image targets according to claim 2, characterized in that: The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes: S2201, for each type of data attribute of the first data set, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first data set; if it is not greater than the first regression discrimination threshold, do not process the data; S2202, after completing S2201 for all data attributes, a preprocessed training data set is obtained.

5. The image target fusion detection processing method according to claim 2, characterized in that: The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module; The feature encoding module includes an input submodule, a first feature encoding submodule and a second feature encoding submodule; The input submodule is connected to the first feature encoding submodule and the second feature encoding submodule; The input submodule includes an input port, a first output port and a second output port, and is used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port; The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module; The first fusion output module is used to output a first fusion signal; The second fusion output module is used to output a second fusion signal; The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the first feature coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module; The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the first feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the first feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the first feature coding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module; The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module; The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module; The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the second feature encoding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module; The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the second feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the second feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the second feature coding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module; The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module; The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module; The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module; The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module; The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module; The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module; The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module; The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module; The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module; The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively; The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively; The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively; The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively; The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module; The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module; The first PReLU activation function module is used to input a first fusion signal; The fifth PReLU activation function module is used to input the second fusion signal; The ninth fully connected layer is used to input image data; The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input into the fusion detection model.

6. The image target fusion detection processing method according to claim 2, characterized in that: The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes: S301, initializing a first training number value; S302, acquiring ultrasonic data from the preprocessed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307; S303, inputting the image data in the preprocessed training data set into the fusion detection model; determining the label data corresponding to the ultrasonic data and the image data input into the fusion detection model as label information; in each execution of S303, the image data input into the fusion detection model is different; S304, using the fusion detection model to process the input information to obtain predicted image target information; S305, calculating the predicted image target information and label information using a loss function to obtain a first difference value; S306, performing an accumulation operation on the first training times; S307, determining whether the first training times value exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308; When the first judgment result is no, determining whether the first difference value is less than a set convergence threshold, and obtaining a second judgment result; When the second judgment result is yes, execute S308; when the second judgment result is no, update the parameters of the feature encoding module and the fusion detection module using the update model, and execute S303; S308, initializing the second training times value; S309, acquiring image data from the preprocessed training data set, and inputting the acquired image data into a fusion detection model; the acquired image data remains unchanged in S310 to S314; S310, inputting the ultrasonic data in the preprocessed training data set into the feature coding module in the fusion detection model; determining the label data corresponding to the image data and ultrasonic data input into the fusion detection model as label information; in each execution of S310, the ultrasonic data input into the feature coding module is different; S311, using the fusion detection model to process the input information to obtain predicted image target information; S312, calculating the predicted image target information and label information using a loss function to obtain a second difference value; S313, performing an accumulation operation on the second training times value; S314, determining whether the second training times value exceeds a preset second training times threshold value, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model; When the third judgment result is no, determining whether the second difference value is less than a set convergence threshold, and obtaining a fourth judgment result; When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

7. The method for fusion detection of image targets according to claim 6, characterized in that: The update model is: θ←θ+v, In the formula, is the first difference value or the second difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, It represents the partial derivative of the variable θ; the i-th input information includes ultrasonic data and image data.

8. A fusion detection and processing device for image targets, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method according to any one of claims 1 to 7.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the image target fusion detection processing method according to any one of claims 1 to 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the fusion detection processing method of the image target according to any one of claims 1 to 7.

Images