A method and device for fusion detection of image targets

Abstract

The present invention discloses a method and device for fusion detection processing of image targets, the method comprising: obtaining a target fusion training data set; the target fusion training data set comprises an image data subset, an ultrasonic data subset and a corresponding label subset; the ultrasonic data subset comprises ultrasonic data; the image data subset comprises image data; preprocessing the target fusion training data set to obtain a preprocessed training data set; using the preprocessed training data set, training a constructed fusion detection model to obtain a trained fusion detection model; using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information.

Description

Technical Field

[0001] The present invention relates to the fields of communications and artificial intelligence, and in particular to a method and device for fusion detection and processing of image targets. Background Art

[0002] With the widespread adoption of unmanned motion platforms and autonomous driving technologies, perception and obstacle avoidance are core challenges. Image-based object detection during motion has become a key technology for this. Efficient and stable object detection ensures route planning and obstacle avoidance for unmanned vehicles, laying the foundation for autonomous driving. However, due to the unique operating characteristics of unmanned vehicles, such as low altitude and highly variable ambient lighting, traditional object detection methods have limitations, making it difficult to achieve real-time and accurate results.

[0003] How to achieve efficient and accurate target detection in images collected in unmanned vehicle working scenarios is a problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention mainly solves the problem of how to achieve efficient and accurate target detection for images collected in unmanned vehicle working scenarios. The present invention discloses a method and device for fusion detection processing of image targets.

[0005] In a first aspect, an embodiment of the present application discloses a method for fusion detection and processing of an image target, comprising:

[0006] S1, obtaining a target fusion training data set; the target fusion training data set includes an image data subset, an ultrasound data subset, and a corresponding label subset; the ultrasound data subset includes ultrasound data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasound data in the image data subset and the ultrasound data subset, respectively;

[0007] S2, preprocessing the target fusion training data set to obtain a preprocessed training data set;

[0008] S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model;

[0009] S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information.

[0010] The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes:

[0011] S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set;

[0012] S22: Perform boundary checking on the first data set to obtain a preprocessed training data set.

[0013] The performing boundary checking on the first data set to obtain a preprocessed training data set includes:

[0014] S221: For each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value and a lower limit value of the value range;

[0015] S222: for each data element of each data attribute of the first data set, determine whether the value of the data element is within the value range according to the value range of the data attribute; if the value is within the value range, do not process the data element; if the value is not within the value range, set the value of the data element to the upper limit value or the lower limit value of the value range that is closest to the data element;

[0016] S223, after completing S222 for the data of all data attributes, a pre-processed training data set is obtained.

[0017] The performing boundary checking on the first data set to obtain a preprocessed training data set includes:

[0018] S2201, for each data attribute of each category in the first data set, perform autoregressive-sliding average modeling, using data collection information of the data as an independent variable and data values ​​of the data as a dependent variable, to obtain regression models for the respective data attributes; using the regression models, perform computational processing on the independent variables to obtain regression data values; determine whether the absolute value of the difference between the regression data values ​​and the corresponding dependent variable values ​​is greater than a set first regression discrimination threshold; if so, delete the data from the first data set; if not, do not process the data;

[0019] S2202: After completing S2201 for all data attributes, a pre-processed training data set is obtained.

[0020] The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module;

[0021] The feature encoding module includes an input submodule, a first feature encoding submodule and a second feature encoding submodule; the input submodule is connected to the first feature encoding submodule and the second feature encoding submodule;

[0022] The input submodule includes an input port, a first output port, and a second output port, and is used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port;

[0023] The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0024] The first fusion output module is configured to output a first fusion signal;

[0025] The second fusion output module is configured to output a second fusion signal;

[0026] The first input module includes an input port, a first output port, and a second output port, and is configured to receive the encoded information output by the first feature encoding submodule and output the encoded information through the first output port and the second output port, respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0027] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the second output port is used to output the K variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the third output port is used to output the V variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0028] The first multi-head attention module includes a first input port, a second input port, a third input port, and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0029] The first addition module is configured to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0030] The second input module includes an input port, a first output port, and a second output port, and is configured to receive the encoded information output by the second feature encoding submodule and output the encoded information through the first output port and the second output port, respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0031] The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the second output port is used to output the K variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the third output port is used to output the V variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module;

[0032] The second multi-head attention module includes a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0033] The second addition module is configured to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module;

[0034] The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing and fusion module, and a one-dimensional deconvolution network module;

[0035] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard code inner product module;

[0036] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; the third The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hadamard code inner product module;

[0037] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard code inner product module;

[0038] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hadamard code inner product module;

[0039] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0040] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0041] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0042] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hadamard code inner product module respectively;

[0043] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0044] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0045] The output end of the first Hadamard inner product module and the output end of the second Hadamard inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0046] The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module;

[0047] The first PReLU activation function module is used to input a first fusion signal;

[0048] The fifth PReLU activation function module is used to input the second fusion signal;

[0049] The ninth fully connected layer is used to input image data;

[0050] The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input to the fusion detection model.

[0051] The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes:

[0052] S301, initializing a first training number value;

[0053] S302, acquiring ultrasonic data from the pre-processed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307;

[0054] S303, inputting the image data in the preprocessed training data set into a fusion detection model; determining label data corresponding to the ultrasonic data and image data input into the fusion detection model, which is label information; in each execution of S303, the image data input into the fusion detection model is different;

[0055] S304, using the fusion detection model to process the input information to obtain predicted image target information;

[0056] S305, calculating the predicted image target information and label information using a loss function to obtain a first difference value;

[0057] S306, performing an accumulation operation on the first training times;

[0058] S307, determining whether the first training times exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308;

[0059] When the first judgment result is no, determining whether the first difference value is less than a set convergence threshold, and obtaining a second judgment result;

[0060] When the second judgment result is yes, execute S308; when the second judgment result is no, use the updated model to update the parameters of the feature encoding module and the fusion detection module, and execute S303;

[0061] S308, initializing the second training number value;

[0062] S309, acquiring image data from the pre-processed training data set, and inputting the acquired image data into the fusion detection model; the acquired image data remains unchanged in S310 to S314;

[0063] S310, inputting the ultrasonic data in the preprocessed training data set into a feature encoding module in a fusion detection model; determining label data corresponding to the image data and ultrasonic data input into the fusion detection model, as label information; in each execution of S310, the ultrasonic data input into the feature encoding module is different;

[0064] S311, using the fusion detection model to process the input information to obtain predicted image target information;

[0065] S312, calculating the predicted image target information and label information using a loss function to obtain a second difference value;

[0066] S313, performing an accumulation operation on the second training times;

[0067] S314, determining whether the second training number value exceeds a preset second training number threshold, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model;

[0068] When the third judgment result is no, determining whether the second difference value is less than a set convergence threshold, and obtaining a fourth judgment result;

[0069] When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

[0070] The update model is:

[0071]

[0072] θ←θ+v,

[0073] Where, is the first difference value or the second difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, It represents the partial derivative of the variable θ; the i-th input information includes ultrasonic data and image data.

[0074] According to a second aspect of an embodiment of the present invention, a device for fusion detection and processing of an image target is disclosed, the device comprising:

[0075] a memory storing executable program code;

[0076] a processor coupled to the memory;

[0077] The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method.

[0078] According to a third aspect of an embodiment of the present invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions. When the computer instructions are called by a computer, the computer instructions are used to execute the image target fusion detection processing method.

[0079] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, which is used to implement the image target fusion detection processing method.

[0080] The beneficial effects of the present invention are:

[0081] The method of the present invention can improve the accuracy of target estimation, reduce the number of iterative calculations, improve calculation efficiency, and at the same time can handle the deformation of moving targets in the field of view, thereby enhancing adaptability to moving objects.

[0082] This invention has achieved significant results in unmanned target detection. It can deduce local details from overall motion trends, enhancing robustness against image deformation and effectively detecting moving targets with dramatic size changes from the perspective of unmanned vehicles. By establishing a neural network model, it can handle varying lighting conditions, eliminating lighting constraints and expanding its scope of application. Motion information is fed back to the control system, enabling effective avoidance of moving obstacles and enhancing the autonomous obstacle avoidance capabilities of unmanned vehicles.

[0083] Experimental results show that the present invention maintains the strengths of traditional methods in dealing with occlusion and small targets, and through technological innovation enables it to handle more complex actual environments, including difficult scenarios such as target deformation and lighting changes, achieving higher robustness, accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 4 is an implementation flow chart of the method of the present invention. DETAILED DESCRIPTION

[0085] In order to better understand the content of the present invention, an embodiment is given here.

[0086] Figure 1 4 is an implementation flow chart of the method of the present invention.

[0087] Aiming at the problem of realizing efficient and accurate target detection in images collected in unmanned working scenarios, the present invention discloses a method and device for image target fusion detection processing.

[0088] In a first aspect, an embodiment of the present application discloses a method for fusion detection and processing of an image target, comprising:

[0089] S1, obtaining a target fusion training data set; the target fusion training data set includes an image data subset, an ultrasound data subset, and a corresponding label subset; the ultrasound data subset includes ultrasound data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasound data in the image data subset and the ultrasound data subset, respectively;

[0090] S2, preprocessing the target fusion training data set to obtain a preprocessed training data set;

[0091] S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model;

[0092] S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information;

[0093] The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes:

[0094] S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set;

[0095] S22: Perform boundary checking on the first data set to obtain a preprocessed training data set.

[0096] The performing boundary checking on the first data set to obtain a preprocessed training data set includes:

[0097] S221: For each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value and a lower limit value of the value range;

[0098] S222: for each data element of each data attribute of the first data set, determine whether the value of the data element is within the value range according to the value range of the data attribute; if the value is within the value range, do not process the data element; if the value is not within the value range, set the value of the data element to the upper limit value or the lower limit value of the value range that is closest to the data element;

[0099] S223, after completing S222 for the data of all data attributes, a pre-processed training data set is obtained.

[0100] The performing boundary checking on the first data set to obtain a preprocessed training data set includes:

[0101] S2201, for each data attribute of each category in the first data set, perform autoregressive-sliding average modeling, using data collection information of the data as an independent variable and data values ​​of the data as a dependent variable, to obtain regression models for the respective data attributes; using the regression models, perform computational processing on the independent variables to obtain regression data values; determine whether the absolute value of the difference between the regression data values ​​and the corresponding dependent variable values ​​is greater than a set first regression discrimination threshold; if so, delete the data from the first data set; if not, do not process the data;

[0102] S2202: After completing S2201 for all data attributes, a pre-processed training data set is obtained.

[0103] The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module;

[0104] The feature coding module includes an input submodule, a first feature coding submodule and a second feature coding submodule; the input submodule is connected to the first feature coding submodule and the second feature coding submodule; the input submodule includes an input port, a first output port and a second output port, and is used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port;

[0105] The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0106] The first fusion output module is configured to output a first fusion signal;

[0107] The second fusion output module is configured to output a second fusion signal;

[0108] The first input module includes an input port, a first output port, and a second output port, and is configured to receive the encoded information output by the first feature encoding submodule and output the encoded information through the first output port and the second output port, respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0109] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the second output port is used to output the K variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the third output port is used to output the V variable after the encoding information output by the first feature encoding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0110] The first multi-head attention module includes a first input port, a second input port, a third input port, and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0111] The first addition module is configured to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0112] The second input module includes an input port, a first output port, and a second output port, and is configured to receive the encoded information output by the second feature encoding submodule and output the encoded information through the first output port and the second output port, respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0113] The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the second output port is used to output the K variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the third output port is used to output the V variable after the encoding information output by the second feature encoding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module;

[0114] The second multi-head attention module includes a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0115] The second addition module is configured to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module;

[0116] The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing and fusion module, and a one-dimensional deconvolution network module;

[0117] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard code inner product module;

[0118] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; the third The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hadamard code inner product module;

[0119] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard code inner product module;

[0120] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hadamard code inner product module;

[0121] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0122] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0123] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0124] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hadamard code inner product module respectively;

[0125] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0126] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0127] The output end of the first Hadamard inner product module and the output end of the second Hadamard inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0128] The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module;

[0129] The first PReLU activation function module is used to input a first fusion signal;

[0130] The fifth PReLU activation function module is used to input the second fusion signal;

[0131] The ninth fully connected layer is used to input image data;

[0132] The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input to the fusion detection model.

[0133] The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes:

[0134] S301, initializing a first training number value;

[0135] S302, acquiring ultrasonic data from the pre-processed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307;

[0136] S303, inputting the image data in the preprocessed training data set into a fusion detection model; determining image target information corresponding to the ultrasonic data and image data input into the fusion detection model as label information; in each execution of S303, the image data input into the fusion detection model is different;

[0137] S304, using the fusion detection model to process the input information to obtain predicted image target information;

[0138] S305, calculating the predicted image target information and label information using a loss function to obtain a difference value;

[0139] S306, performing an accumulation operation on the first training times;

[0140] S307, determining whether the first training times exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308;

[0141] When the first judgment result is no, determining whether the difference value is less than a set convergence threshold, and obtaining a second judgment result;

[0142] When the second judgment result is yes, execute S308; when the second judgment result is no, use the updated model to update the parameters of the feature encoding module and the fusion detection module, and execute S303;

[0143] S308, initializing the second training number value;

[0144] S309, acquiring image data from the pre-processed training data set, and inputting the acquired image data into the fusion detection model; the acquired image data remains unchanged in S310 to S314;

[0145] S310, inputting the ultrasonic data in the preprocessed training data set into a feature encoding module in a fusion detection model; determining image target information corresponding to the image data and ultrasonic data input into the fusion detection model as label information; in each execution of S310, the ultrasonic data input into the feature encoding module is different;

[0146] S311, using the fusion detection model to process the input information to obtain predicted image target information;

[0147] S312, calculating the predicted image target information and label information using a loss function to obtain a difference value;

[0148] S313, performing an accumulation operation on the second training times;

[0149] S314, determining whether the second training number value exceeds a preset second training number threshold, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model;

[0150] When the third judgment result is no, determining whether the difference value is less than a set convergence threshold, and obtaining a fourth judgment result;

[0151] When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

[0152] The update model is:

[0153]

[0154] θ←θ+v,

[0155] Where, is the difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, The i-th input information includes ultrasonic data and image data.

[0156] The loss function may be a cross entropy loss function.

[0157] Whether the difference value meets the convergence condition refers to whether the difference value is less than the preset error threshold value; when the difference value is less than the preset error threshold value, it is judged that the difference value meets the convergence condition; when the difference value is greater than the preset error threshold value, it is judged that the difference value does not meet the convergence condition.

[0158] The autoregressive-moving average modeling can be implemented using the ARMA method;

[0159] The first feature encoding submodule is implemented using the ShuffleNetV2 network;

[0160] The second feature encoding submodule is implemented using a one-dimensional convolutional neural network;

[0161] Outlier elimination includes filling missing values, smoothing noise data, and smoothing or deleting outlier points. Smoothing noise data involves first identifying noise data and then smoothing it based on the preceding and following data. Noise data is defined as values ​​that are less than the sensor's sensitivity or greater than the sensor's upper limit. Kalman filtering can be used to identify outliers. The imputation value for missing values ​​can be determined by averaging the measured values ​​within a certain sampling interval before and after the missing value.

[0162] The autoregressive-moving average modeling can be achieved by performing curve fitting on the curve to be approximated using a function approximation method.

[0163] The curve fitting of the curve to be approximated by the function approximation method can be performed using the best consistent linear approximation method. The best consistent approximation polynomial f(Ix) is expressed as:

[0164] f(Ix)=α P1 (Ix) P1 +α P1-1 (Ix) P1-1 +…+α2(Ix) 2 +α1(Ix)+α0,

[0165] Wherein, P1 is the order of the best consistent approximation polynomial f(Ix), α0, α1, α2,…, α P1 are the coefficients of the optimal consistent approximation polynomial f(Ix);

[0166] The data collection information includes time information, space information or capability type information corresponding to the data collection; when performing data fitting, any of the above types of information can be used as an independent variable, and the corresponding data value can be used as a dependent variable.

[0167] The first feature encoding submodule is implemented using the ShuffleNetV2 network;

[0168] The second feature encoding submodule is implemented using a one-dimensional convolutional neural network;

[0169] The trained fusion detection model is used to process the collected image data and ultrasonic data to obtain target information, including:

[0170] Using the trained fusion detection model, the collected image data and ultrasonic data are processed to obtain a target occurrence probability value;

[0171] Determine whether the target occurrence probability value is greater than a preset probability threshold value. If so, set the target information as containing the target; if not, set the target information as not containing the target.

[0172] The target information is used to indicate whether the acquired image data and ultrasonic data contain a target;

[0173] The ultrasonic data is ultrasonic training data constructed by collecting data using an ultrasonic sensor installed on an unmanned motion platform;

[0174] The image data is image training data constructed by collecting data using an image sensor installed on an unmanned motion platform;

[0175] According to a second aspect of an embodiment of the present invention, a device for fusion detection and processing of an image target is disclosed, the device comprising:

[0176] a memory storing executable program code;

[0177] a processor coupled to the memory;

[0178] The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method.

[0179] According to a third aspect of an embodiment of the present invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions. When the computer instructions are called by a computer, the computer instructions are used to execute the image target fusion detection processing method.

[0180] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, which is used to implement the image target fusion detection processing method.

[0181] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A fusion detection processing method for image targets, characterized in that: include: S1, obtain the target fusion training data set; The target fusion training data set includes an image data subset, an ultrasound data subset and a corresponding label subset; The ultrasonic data subset includes ultrasonic data; the image data subset includes image data; any label data in the label subset has corresponding image data and ultrasonic data in the image data subset and the ultrasonic data subset, respectively; S2, preprocessing the target fusion training data set to obtain a preprocessed training data set; S3, using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model; S4, using the trained fusion detection model, processing the collected image data and ultrasonic data to obtain target information; The preprocessing of the target fusion training data set to obtain a preprocessed training data set includes: S21, performing outlier elimination processing on the target fusion training data set to obtain a first data set; S22, performing boundary check processing on the first data set to obtain a preprocessed training data set; The fusion detection model includes: a feature encoding module, a fusion encoding module and a fusion detection module; The feature encoding module includes an input submodule, a first feature encoding submodule and a second feature encoding submodule; the input submodule is connected to the first feature encoding submodule and the second feature encoding submodule; The input submodule includes an input port, a first output port and a second output port, and is used to receive ultrasonic data and image data, send the image data to the first feature coding submodule through the first output port, and send the ultrasonic data to the second feature coding submodule through the second output port; The fusion encoding module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module; The first fusion output module is used to output a first fusion signal; The second fusion output module is used to output a second fusion signal; The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the first feature coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module; The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the first feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the first feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the first feature coding submodule is self-attention encoded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the first multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the first multi-head attention module; The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module; The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module; The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the second feature encoding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the second addition module; The second self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the second feature coding submodule is self-attention encoded; the second output port is used to output the K variable after the coding information output by the second feature coding submodule is self-attention encoded; the third output port is used to output the V variable after the coding information output by the second feature coding submodule is self-attention encoded; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the second multi-head attention module; The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module; The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module; The fusion detection module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module; The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module; The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module; The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module; The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module; The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module; The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module; The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively; The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively; The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively; The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively; The output end of the first Hadamard inner product module and the output end of the second Hadamard inner product module are respectively connected to the input end of the fifth splicing and fusion module; The output end of the fifth splicing and fusion module is connected to the input end of the one-dimensional deconvolution network module; The first PReLU activation function module is used to input a first fusion signal; The fifth PReLU activation function module is used to input the second fusion signal; The ninth fully connected layer is used to input image data; The one-dimensional deconvolution network module is used to output image target information; the image target information represents the probability of the target appearing in the image data input into the fusion detection model.

2. The method for fusion detection of image targets according to claim 1, characterized in that: The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes: S221, for each data attribute of the first data set, preset a corresponding value range; the value range includes an upper limit value of the value range and a lower limit value of the value range; S222, for each data of each data attribute of the first data set, judging whether the value of the data is within the value range according to the value range of the data attribute; if the data is within the value range, not processing the data; if the data is not within the value range, setting the value of the data to the upper limit value or the lower limit value of the value range that is closest to the data; S223, after completing S222 for the data of all data attributes, a preprocessed training data set is obtained.

3. The image target fusion detection processing method according to claim 1, characterized in that: The performing boundary checking processing on the first data set to obtain a preprocessed training data set includes: S2201, for each type of data attribute of the first data set, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first data set; if it is not greater than the first regression discrimination threshold, do not process the data; S2202, after completing S2201 for all data attributes, a preprocessed training data set is obtained.

4. The image target fusion detection processing method according to claim 1, characterized in that: The method of using the pre-processed training data set to train the constructed fusion detection model to obtain a trained fusion detection model includes: S301, initializing a first training number value; S302, acquiring ultrasonic data from the preprocessed training data set, and inputting the acquired ultrasonic data into a feature encoding module of a fusion detection model; the acquired ultrasonic data remains unchanged in S301 to S307; S303, inputting the image data in the preprocessed training data set into the fusion detection model; determining the label data corresponding to the ultrasonic data and the image data input into the fusion detection model as label information; in each execution of S303, the image data input into the fusion detection model is different; S304, using the fusion detection model to process the input information to obtain predicted image target information; S305, calculating the predicted image target information and label information using a loss function to obtain a first difference value; S306, performing an accumulation operation on the first training times; S307, determining whether the first training times value exceeds a preset first training times threshold, and obtaining a first determination result; when the first determination result is yes, executing S308; When the first judgment result is no, determining whether the first difference value is less than a set convergence threshold, and obtaining a second judgment result; When the second judgment result is yes, execute S308; when the second judgment result is no, update the parameters of the feature encoding module and the fusion detection module using the update model, and execute S303; S308, initializing the second training number value; S309, acquiring image data from the preprocessed training data set, and inputting the acquired image data into a fusion detection model; the acquired image data remains unchanged in S310 to S314; S310, inputting the ultrasonic data in the preprocessed training data set into the feature coding module in the fusion detection model; determining the label data corresponding to the image data and ultrasonic data input into the fusion detection model as label information; in each execution of S310, the ultrasonic data input into the feature coding module is different; S311, using the fusion detection model to process the input information to obtain predicted image target information; S312, calculating the predicted image target information and label information using a loss function to obtain a second difference value; S313, performing an accumulation operation on the second training times value; S314, determining whether the second training times value exceeds a preset second training times threshold value, and obtaining a third determination result; when the third determination result is yes, completing the training process, and obtaining a trained fusion detection model; When the third judgment result is no, determining whether the second difference value is less than a set convergence threshold, and obtaining a fourth judgment result; When the fourth judgment result is yes, the training process is completed to obtain a trained fusion detection model; when the fourth judgment result is no, the parameters of the fusion encoding module and the fusion detection module are updated using the updated model, and S310 is executed.

5. The image target fusion detection processing method according to claim 4, characterized in that: The update model is: θ←θ+v, In the formula, is the first difference value or the second difference value calculated after the i-th input information in the preprocessed training data set is input into the fusion detection model, v is the parameter update value, θ is the parameter of the network module that needs to be updated, η is the preset initial parameter learning rate, α is the preset momentum angle parameter, 0≤α≤π / 4, It represents the partial derivative of the variable θ; the i-th input information includes ultrasonic data and image data.

6. A fusion detection and processing device for image targets, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the image target fusion detection processing method according to any one of claims 1 to 5.

7. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the image target fusion detection processing method according to any one of claims 1 to 5.

8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the fusion detection processing method of the image target as described in any one of claims 1 to 5.

Images