Human body target accurate sensing method and system for water surface emergency rescue

Abstract

The invention discloses a human body target accurate sensing method and system for water surface emergency rescue, which is used in the fields of water surface exploration, water surface rescue, water surface early warning and the like, and comprises the following steps: S1, setting an exploration area range constraint, and shooting a video sequence image of a target; s2, modeling is carried out on an image human body target recognition problem, and target detection is completed; s3, performing algorithm construction on the video sequence image to complete target positioning; s4, target behavior recognition is carried out, and target state evaluation is completed; and S5, dynamically updating the state quantity of the target, and effectively probing the human body target in the region. A double-layer optimization model constructed by the invention obtains a fused image taking both detection precision and visual effect into consideration, and realizes water surface target detection; constructing a robust block matching algorithm and a polynomial correction algorithm to realize accurate positioning of a water surface target; and fusing the feature maps under infrared and visible light to identify abnormal behaviors, realizing state evaluation of the person in distress, outputting a target state quantity, and realizing accurate perception of a water surface human body target.

Description

Technical Field

[0001] The present invention relates to the fields of water surface exploration, water surface rescue and water surface early warning, and is a human target precise perception method and system for water surface emergency rescue. Background Art

[0002] On July 13, 2023, the latest statistics of the "Drone Rescue Map" released by DJI showed that more than 1,000 people have been rescued in rescue missions directly participated by drones worldwide. This shows that drones have become an important scientific and technological force in the new emergency rescue system in the world. On December 29, 2023, the Ministry of Emergency Management and the Ministry of Industry and Information Technology issued the "Guiding Opinions on Accelerating the Development of Emergency Robots", which pointed out that in response to the needs of multifunctional integration of emergency robots, a batch of high-performance payloads should be developed, and the research on environmental perception and target reconnaissance payloads should be strengthened to enhance the robot's ability to perform efficient search operations. Due to its excellent maneuverability and low cost, drones have become an important development aspect of emergency search and rescue robots. Strengthening drone reconnaissance and search and rescue technology is not only an important cornerstone for promoting the modernization of emergency capabilities, but also an important guarantee for the safety of life and property of the people across the country. Traditional single image sensors often find it difficult to independently extract sufficient and sufficient target information from the scene. Based on the dual-light fusion carried by drones, infrared and visible light image fusion technology is used to extract and fuse the features of infrared images and visible light images, which can obtain an image with significant targets and rich texture details. The precise perception of human targets on the water surface is divided into two basic modules: target detection and positioning. On this basis, a state assessment module is added. The present invention realizes the above functions based on the image containing rich information and feature points of dual-light fusion, and completes the precise perception of human targets on the water surface. Summary of the invention

[0003] The present invention proposes a human target accurate perception method and system for surface emergency rescue. First, the range constraint of the area to be explored is set, and a video sequence image is taken of the suspected target; then a model is established for the image human target recognition problem to complete target detection; then an algorithm is constructed for the video sequence image to complete the positioning of the target; then a multimodal feature fusion module is set for behavior recognition to complete the target state evaluation; finally, the state quantity of the target is dynamically updated to achieve effective detection of the human target in the area.

[0004] In order to achieve the above purpose, the technical solution provided by the present invention is as follows:

[0005] A human target accurate perception method for surface emergency rescue, the method comprising:

[0006] S1. Set the range constraints of the area to be explored and take video sequence images of the suspected target;

[0007] S2. Build a model for human target recognition in images and complete target detection;

[0008] S3, construct an algorithm for the video sequence images and complete the positioning of the target;

[0009] S4, setting a multimodal feature fusion module to perform target behavior recognition and complete target state evaluation;

[0010] S5. Dynamically update the state of the target to effectively detect human targets in the area.

[0011] To further illustrate, the step S1 is specifically as follows:

[0012] S11. Set an irregular water area with a boundary L;

[0013] S12. Set the total number of suspected targets n t , and initialize the state of each suspected target, including the suspected target position (T x_i ,T y_i ), the number of suspected targets to be detected T r_i , initial detection angle of suspected target T a_i and the suspected target detection change threshold (T rc_i ,T ac_i ), where i represents the i-th suspected target, and its value range is [1, nt];

[0014] S13. Shooting infrared and visible light video sequence images based on an airborne dual-light fusion camera.

[0015] To further illustrate, the step S2 is specifically as follows:

[0016] S21. In order to better describe the joint problem of image fusion and target detection, the video sequence images are regarded as a set, and the problem setting of human target detection based on the two-layer optimization model is explained.

[0017] S22, Assume that the infrared image, visible light image and fused image are all grayscale images with sizes of m×n, and use column vectors x, y and u∈R mn×1 It indicates that a two-layer optimization model is constructed;

[0018]

[0019] To detect a specific training loss, u * is the output result image, ψ is a learnable parameter ω d detection network; f(·) is an energy-based fidelity term that contains the fused image u, source images x and y, g T (·) and gD (·) are two feasibility constraints defined on infrared and visible light, respectively.

[0020] S23, expand the model into a dual adversarial fusion network, consisting of a generator and two object-aware discriminators and a common detection network;

[0021] S24, the generator generates a fused image, the target discriminator evaluates the intensity consistency between the infrared image and the fused image, and the detail discriminator distinguishes the gradient distribution between the visible light image and the fused image;

[0022] S25. Output the detection-oriented fusion image and import it into the detection network of YOLOv5 to complete the target detection.

[0023] To further illustrate, the step S3 is specifically as follows:

[0024] S31, extracting a datum-digital orthophoto map (datum-DOM) from a region-digital orthophoto map (region-DOM) according to the POS data of the video frame, and replacing the region-DOM with the datum-DOM as a new matching region of the video frame;

[0025] S32. Unify the spatial resolution of datum-DOM and video frames. The ratio of datum-DOM is:

[0026]

[0027] Indicates scaling the size of datum-DOM, img D Indicates the size of the original datum-DOM, Scale D =gsd D / gsd F , gsd D is the spatial resolution of the original datum-DOM, gsd F is the spatial resolution of the video frame;

[0028] S33, according to the similarity between the video frame image and the datum-DOM in the RGB color space, find the best matching area of ​​the video frame on the datum-DOM. Assume (x L1 ,y L1 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on the datum-DOM in the RGB color space, (x L1 ,y L1 ) can be obtained:

[0029]

[0030] and are the intensities of the R, G, and B channels of the video frame, respectively; and are the intensities of the datum-DOM channels R, G, and B respectively; (x, y) is the pixel coordinate in the video frame, x = (1, 2, ..., N LF ), y=(1,2,…,N WF ), (△x1,△y1) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x1=(1,2,…,N LD -N LF ), △y1=(1,2,…,N WD -N WF );N LF and N WF are the length and width of the video frame, in pixels; N LD and N WD are the length and width of datum-DOM, in pixels; N F is the total number of pixels in the video frame, N F =N LF N WF ;

[0031] S34. According to the similarity between the two images in the gradient amplitude space, find the best matching area of ​​the video frame on the datum-DOM. Assume (x L2 ,y L2 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on datum-DOM in the gradient amplitude space, (x L2 ,y L2 ) can be obtained:

[0032]

[0033] F x (x,y) and F y (x, y) is the partial derivative of the video frame in the x and y directions respectively; D x (△x2+x,△y2+y) and D y (△x2+x,△y2+y) are the first-order partial derivatives of datum-DOM in the x and y directions respectively; (x,y) are the pixel coordinates in the video frame, x=(1,2,…,N LF ), y=(1,2,…,N WF ), (△x2,△y2) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x2=(1,2,…,NLD -N LF ), △y2=(1,2,…,N WD -N WF );N LF and N Wf are the length and width of the video frame, in pixels; N LD and N WD They are the length and width of datum-DOM respectively, in pixels;

[0034] S35, construct a robust block matching algorithm that considers both the color difference and texture complexity of the video frame, set the symbol that comprehensively represents the color difference amplitude and texture complexity of the video frame as TH, and select a threshold of 20 to determine TH. (x L ,y L ) is the coordinate of the upper left corner of the best matching area obtained by the block matching algorithm, which can be calculated as:

[0035]

[0036] TH=∑ x,y |I F (x,y)-I D (△x2+x,△y2+y)| / N F

[0037] ω L is the weight, r represents (x L1 ,y L1 ) and (x L2 ,y L2 ), N LF and N WF are the length and width of the video frame, in pixels. L ,y L ,N LF ,N WF ) extracting a block matching DOM from the datum-DOM, that is, the best matching area of ​​the video frame on the datum-DOM;

[0038] S36, constructing polynomials of the video frame and the block matching DOM, and evenly dividing the gradient amplitude image of the video frame and the gradient amplitude image of the block matching DOM into n×n local areas, respectively, and each local area is represented by a second-order polynomial as follows:

[0039]

[0040] and are the local area intensities of the i-th column and the j-th column in the two images, i = (1, ..., n), j = (1, ..., n); X I =(x I ,y I ) T , X Π =(x Π ,y Π ) T , T represents the transpose of the matrix (vector), (x I ,y I ) and (x Π ,y Π ) are the pixel coordinates of the local areas in the two images, T Ι and T Π are the generation time of the video frame and block matching DOM respectively;

[0041] A Ι and A Π are the second-order coefficient matrices of their polynomials respectively;

[0042] B I and B Π are the first-order coefficient vectors of their polynomials respectively;

[0043] C Ι and C Π are the scalars of their polynomials respectively; are multiple parameters of the polynomial;

[0044] S37. Construct a differential-difference polynomial, which can be further rewritten using the first-order derivative Taylor expansion d of ΔX as:

[0045]

[0046] ΔX is the small motion displacement from the local area of ​​the video frame to the corresponding local area of ​​the block matching DOM, f F (X I ,T I ) and f D (X Π ,T Π ) are the intensities of the corresponding local areas, X I and X Π are the pixel coordinates of the local area, T Ι and T Π are the generation time of the video frame and block matching DOM respectively;

[0047] are multiple parameters of the polynomial. X Π =X I +ΔX; let d=0 and further get the equation of ΔX:

[0048] A ΔX ΔX=L ΔX

[0049]

[0050] S38. Establish a second-order polynomial expression for ΔX:

[0051]

[0052] (x f ,y f ) are the coordinates of the local area in the video frame, a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5 are polynomial parameters; construct an accurate correction equation:

[0053] At=L

[0054] x f and f is the column and row number of a local area in the video frame; t = (a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5) T , t is an unknown parameter vector to be solved;

[0055] S39. Obtain the geodetic coordinates of the pixels in the video frame according to the calculated optimal estimate t.

[0056] To further illustrate, the step S4 is specifically as follows:

[0057] S41, input the infrared image and visible light image taken by airborne dual-light fusion, and extract each pair of features from the infrared image and the visible light image: (thermal features on RGB stream) and (visible features on RGB stream), (thermal signature on the moving flow) and (visible features on the motion flow);

[0058] S42. Given a feature map, a fusion module is designed using a weighted assembly strategy to calculate the responses at all pixels as a weight matrix:

[0059]

[0060] FU S(·) is feature fusion, W FUs are the learnable parameters of the feature extractor, represents the thermal characteristics of the flow S, represents the visible light characteristics of flow S, and is the weighted matrix of thermal and visible light variations of stream S after normalization by the aggregation function Agg(·); the multimodal feature fusion (MFF) module examines two Agg(·) functions: sigmoid and softmax, which are applied to the score matrix and The corresponding index (i, j, k) of each element in (i, j, k are used for height, width and time axis respectively), specifically:

[0061]

[0062] It represents the fusion weight assigned by stream S to the corresponding pixel in a given feature map. The superscript t represents infrared (heat) and v represents visible light.

[0063] Calculate the fusion feature f RGB (characteristics of RGB stream) and f Motion (Characteristics of motion flow) formula:

[0064]

[0065] represents the weight of the thermal feature in the RGB stream, represents the weight of the visible features in the RGB stream, represents the weight of the thermal feature in the motion flow, Represents the weight of the visible feature in the motion flow; the superscript t indicates that it corresponds to infrared (heat), and v indicates that it corresponds to visible light.

[0066] S43. Design the Classification module, i.e., the classification module, to generate the embedding vector z of the RGB stream and motion stream RGB and z Motion :

[0067]

[0068]

[0069] C RGB (·), C Motion (·) indicates the classification of RGB stream and motion stream respectively, and is the training parameter of the classification module, z RGB and z MotionThe dimension is the number of categories. The value of each instance represents the prediction score of the corresponding category. The SoftMax function is used as the normalization function of the embedding vector to obtain the prediction score of the RGB stream. and the prediction scores of motion flow

[0070]

[0071] S44. Combine RGB and Motion embedding vectors to generate z Fus , and generate predictions for the Fusion stream:

[0072] z Fus =z RGB +z Motion

[0073]

[0074] represents the prediction score of the fused stream;

[0075] S45, final classification loss L C is the sum of the losses on each flow:

[0076]

[0077] L C =L Fus +L RGB +L Motion

[0078] FL(·) is the focal loss function, and the combined embedding vector z Fus Directly affected by RGB and z Motion In order to reduce the difference between the negative and positive effects of correlation, the embedding loss L is designed emb :

[0079] L emb =MSE(z RGB ,z Motion )

[0080] MSE(·) is the mean square error loss function. The MFF network minimizes the combined loss L, which is the sum of the classification loss and the embedding loss:

[0081] L=L C +L emb

[0082] The combined loss helps the MFF model strike a balance between representing RGB or optical flow input and classification loss;

[0083] S46. Analyze the target human behavior based on the fused feature map and assign a state classification label to each target.

[0084] To further illustrate, the step S5 is specifically as follows:

[0085] S51, outputting human target information and its position information and status information at a specific moment;

[0086] S52, dynamically update the location information and status information according to the time sequence, and return it to the ground dispatcher to achieve full process supervision;

[0087] S53, ground dispatch to rescue the human target, and the human target detection mission is completed.

[0088] The present invention also proposes a human target precision perception system for surface emergency rescue, comprising a visual sensor module and an information processing module;

[0089] The visual sensor module is used to capture infrared images and visible light images of video sequence images;

[0090] The information processing module completes the detection, positioning and state estimation of the target according to the video sequence images taken by the visual sensor module, which is specifically achieved through the aforementioned steps S1-S4.

[0091] Beneficial effects of the present invention:

[0092] 1. The present invention constructs a two-layer optimization model based on the video sequence images shot by airborne dual-light fusion to obtain a fused image that takes into account both detection accuracy and visual effects, thereby realizing surface target detection.

[0093] 2. Build a robust block matching algorithm and an accurate polynomial correction algorithm to achieve accurate positioning of surface targets.

[0094] 3. Construct a multimodal feature fusion method to fuse the feature maps under infrared and visible light to identify abnormal behaviors, realize the status assessment of people in distress, and output the status quantity of human targets to complete the accurate perception of human targets on the water surface.

[0095] 4. The present invention can be applied to the fields of water surface exploration, water surface rescue and water surface early warning, and has a wide range of applications, such as using drones to search for human targets in flood disasters in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 The overall process diagram of the present invention;

[0097] Figure 2 Flow chart of the target positioning algorithm of the present invention;

[0098] Figure 3 The state assessment flow chart of the present invention; DETAILED DESCRIPTION

[0099] The present invention will be further described below in conjunction with the accompanying drawings.

[0100] like Figure 1 As shown, the present invention is a method for accurately sensing human targets for surface emergency rescue, comprising the following steps:

[0101] S1. Set the range constraints of the area to be explored and take video sequence images of the suspected target;

[0102] S2. Build a model for human target recognition in images and complete target detection;

[0103] S3, construct an algorithm for the video sequence images and complete the positioning of the target;

[0104] S4, setting the multimodal feature fusion module for behavior recognition and completing target state evaluation;

[0105] S5. Dynamically update the state of the target to effectively detect human targets in the area.

[0106] To further illustrate, the step S1 is specifically as follows:

[0107] Set the range constraints of the area to be explored and take video sequence images of suspected targets;

[0108] S11. Set an irregular water area with a boundary L;

[0109] S12. Set the total number of suspected targets n t , and initialize the state of each suspected target, including the suspected target position (T x_i ,T y_i ), the number of suspected targets to be detected T r_i , initial detection angle of suspected target T a_i and the suspected target detection change threshold (T rc_i ,T ac_i ), where i represents the i-th suspected target, and its value range is [1,n t ];

[0110] S13. Shooting infrared and visible light video sequence images based on airborne dual-light fusion.

[0111] To further illustrate, the step S2 is specifically as follows:

[0112] Establish a model for human target recognition in images and complete target detection;

[0113] S21. In order to better describe the joint problem of image fusion and target detection, the video sequence images are regarded as a set, and the problem setting of human target detection based on the two-layer optimization model is explained.

[0114] S22. Assume that the infrared, visible light and fused images are all grayscale images with sizes of m×n, and use column vectors x, y and u∈R mn×1 , construct a two-layer optimization model;

[0115]

[0116] To detect a specific training loss, ψ is a learnable parameter ω d detection network; f(·) is an energy-based fidelity term that contains the fused image u, source images x and y, g T (·) and g D (·) are two feasibility constraints defined on infrared and visible light, respectively.

[0117] S23, the model is expanded to a dual adversarial fusion network, introducing an adversarial game consisting of a generator and two discriminators;

[0118] S24, the generator provides a fused image, the target discriminator evaluates the intensity consistency between the infrared target and the fused target, and the detail discriminator distinguishes the gradient distribution of the visible light target and the fused target;

[0119] S25: Output the fused image for detection and complete the target detection according to the YOLOv5 algorithm.

[0120] To further illustrate, the step S3 is specifically as follows:

[0121] The algorithm is constructed for the video sequence images to complete the positioning of the target. The process is as follows: Figure 2 As shown;

[0122] S31, extracting datum-DOM from region-DOM according to the POS data of the video frame, and replacing region-DOM with the datum-DOM as a new matching region of the video frame;

[0123] S32. Unify the spatial resolution of datum-DOM and video frames. The ratio of datum-DOM is:

[0124]

[0125] Indicates scaling the size of datum-DOM, img D Indicates the size of the original datum-DOM, Scale D =gsdD / gsd F , gsd D is the spatial resolution of the original datum-DOM, gsd F is the spatial resolution of the video frame;

[0126] S33. According to the similarity between the two images in the RGB color space, find the best matching area of ​​the video frame on the datum-DOM. Assume (x L1 ,y L1 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on the datum-DOM in the RGB color space, (x L1 ,y L1 ) can be obtained:

[0127]

[0128] and are the intensities of the R, G, and B channels of the video frame, respectively; and are the intensities of the datum-DOM channels R, G, and B respectively; (x, y) is the pixel coordinate in the video frame, x = (1, 2, ..., N LF ), y=(1,2,…,N WF ), (△x1,△y1) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x1=(1,2,…,N LD -N LF ), △y1=(1,2,…,N WD -N WF );N LF and N WF are the length and width of the video frame, in pixels; N LD and N WD are the length and width of datum-DOM, in pixels; N F is the total number of pixels in the video frame, N F =N LF N WF ;

[0129] S34. According to the similarity between the two images in the gradient amplitude space, find the best matching area of ​​the video frame on the datum-DOM. Assume (x L2 ,y L2 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on datum-DOM in the gradient amplitude space, (x L2 ,y L2 ) can be obtained:

[0130]

[0131] F x (x,y) and F y (x, y) is the partial derivative of the video frame in the x and y directions respectively; D x (△x2+x,△y2+y) and D y (△x2+x,△y2+y) are the first-order partial derivatives of datum-DOM in the x and y directions respectively; (x,y) are the pixel coordinates in the video frame, x=(1,2,…,N LF ), y=(1,2,…,N WF ), (△x2,△y2) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x2=(1,2,…,N LD -N LF ), △y2=(1,2,…,N WD -N WF );N LF and N WF are the length and width of the video frame, in pixels; N LD and N WD They are the length and width of datum-DOM respectively, in pixels;

[0132] S35, construct a robust block matching algorithm that considers both the color difference and texture complexity of the video frame, propose a symbol TH that comprehensively represents the color difference amplitude and texture complexity of the video frame, and select a threshold of 20 to judge TH. (x L ,y L ) is the coordinate of the upper left corner of the best matching area obtained by the block matching algorithm, which can be calculated as:

[0133]

[0134] TH=∑ x,y |I F (x,y)-I D (△x2+x,△y2+y)| / N F

[0135] ω L is the weight, r represents (x L1 ,y L1 ) and (x L2 ,y L2 ), N LF and N WF are the length and width of the video frame, in pixels. L ,y L ,N LF,N WF ) extracting a block matching DOM, i.e., a best matching area of ​​the video frame on the datum-DOM, from the datum-DOM;

[0136] S36, constructing polynomials of the video frame and the block matching DOM, and evenly dividing the gradient amplitude image of the video frame and the gradient amplitude image of the block matching DOM into n×n local areas, respectively, and each local area is represented by a second-order polynomial as follows:

[0137]

[0138] and are the local area intensities of the i-th column and the j-th column in the two images, i = (1, ..., n), j = (1, ..., n); X I =(x I ,y I ) T , X Π =(x Π ,y Π ) T , T represents the transpose of the matrix (vector), (x I ,y I ) and (x Π ,y Π ) are the pixel coordinates of the local areas in the two images, T Ι and T Π are the generation time of the video frame and block matching DOM respectively;

[0139] A Ι and A Π are the second-order coefficient matrices of their polynomials respectively; B I and B Π are the first-order coefficient vectors of their polynomials respectively; C Ι and C Π are the scalars of their polynomials respectively; are multiple parameters of the polynomial;

[0140] S37. Construct a differential-difference polynomial, which can be further rewritten using the Taylor expansion of the first-order derivative of ΔX as:

[0141]

[0142] f F (X I ,T I ) and f D (X Π ,TΠ ) are the intensities of the corresponding local areas, X I and X Π are the pixel coordinates of the local area, T Ι and T Π are the generation time of the video frame and block matching DOM respectively;

[0143] are multiple parameters of the polynomial. X Π =X I +ΔX; let d=0 and further get the equation of ΔX:

[0144] A ΔX ΔX=L ΔX

[0145]

[0146] The second-order polynomial of S38 and ΔX is expressed as

[0147]

[0148] (x f ,y f ) are the coordinates of the local area in the video frame, a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5 are polynomial parameters; construct an accurate correction equation:

[0149] At=L

[0150] x f and f is the column and row number of a local area in the video frame; t = (a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5) T , t is an unknown parameter vector to be solved;

[0151] S39, further illustrating obtaining the geodetic coordinates of the pixels in the video frame according to the calculated optimal estimate t.

[0152] The step S4 is specifically as follows:

[0153] Set up a multimodal feature fusion module for behavior recognition and complete target state evaluation. The process is as follows: Figure 3 As shown;

[0154] S41, input the infrared image and visible light image taken by airborne dual-light fusion, and extract each pair of features from the thermal input and visible light input: and and

[0155] S42. Given a feature map, a fusion module is designed using a weighted assembly strategy to calculate the responses at all pixels as a weight matrix:

[0156]

[0157] and is the weighted matrix of thermal and visible changes of stream S after normalization by the aggregation function Agg(·); the multimodal feature fusion (MFF) module examines two Agg(·) functions, sigmoid and softmax, which are applied to the score matrix and Each element in at the corresponding index (i,j,k) (for height, width and time axis) is:

[0158]

[0159] Calculate the fusion feature f RGB and f Motion The formula is:

[0160]

[0161] S43. Design the Classification module to generate the embedding vector z RGB and z Motion :

[0162]

[0163] and is the parameter of the classification module, z RGB and z Motion The dimension is the number of categories. The value of each instance represents the prediction score of the corresponding category. The SoftMax function is used as the normalization function of the embedding vector to obtain the prediction result:

[0164]

[0165] S44. Combine RGB and Motion embedding vectors to generate z Fus , and generate predictions for the Fusion stream:

[0166] z Fus =z RGB +z Motion

[0167]

[0168] S45, final classification loss L C is the sum of the losses on each flow:

[0169]

[0170] L C =L Fus +L RGB +L Motion

[0171] Combined embedding vector z Fus Directly affected by RGB and z Motion To reduce the difference between the negative and positive effects of correlation, L emb :

[0172] L emb =MSE(z RGB ,z Motion )

[0173] MSE(·) is the mean square error loss function. The MFF network minimizes the combined loss, which is the sum of the classification loss and the embedding loss:

[0174] L=L C +L emb

[0175] The combined loss helps the MFF model strike a balance between representing RGB or optical flow input and classification loss;

[0176] S46. Complete the recognition of abnormal human behavior and label each target with a status classification label.

[0177] To further illustrate, the step S5 is specifically as follows:

[0178] Dynamically update the state of the target to effectively detect human targets in the area;

[0179] S51, outputting human target information and its position information and status information at a specific moment;

[0180] S52, dynamically update the location information and status information according to the time sequence, and return it to the ground dispatcher to achieve full process supervision;

[0181] S53, ground dispatch to rescue the human target, and the human target detection mission is completed.

[0182] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A human target accurate perception method for surface emergency rescue, characterized in that: These include: S1. Set the range constraints of the area to be explored and obtain the video sequence images of the suspected target; S2, modeling human target recognition in images to complete target detection; S3. Construct an algorithm for the video sequence images to complete target positioning.

2. According to claim 1, a human target accurate perception method for surface emergency rescue is characterized in that: The specific implementation of S1 includes: S11. Set an irregular water area with a boundary L; S12. Set the total number of suspected targets n t , and initialize the state of each suspected target, including the suspected target position (T x_i ,T y_i ), the number of suspected targets to be detected T r_i , initial detection angle of suspected target T a_i and the suspected target detection change threshold (T rc_i ,T ac_i ), where i represents the i-th suspected target, and its value range is [1,n t ]; S13. Shooting infrared and visible light video sequence images based on an airborne dual-light fusion camera.

3. The method for accurately sensing human targets for surface emergency rescue according to claim 1 is characterized in that: The specific implementation of S2 includes: S21. Consider the video sequence images as a set and establish a human target detection algorithm based on a two-layer optimization model; S22, Assume that the infrared, visible light and fused images are all grayscale images with sizes of m×n, and use column vectors x, y and u∈R mn×1 , construct a two-layer optimization model; To detect a specific training loss, ψ is a learnable parameter ω d detection network; f(·) is an energy-based fidelity term that contains the fused image u, source images x and y, g T (·) and g D (·) are two feasibility constraints defined on infrared and visible light, respectively; S23, expand the model to a dual adversarial fusion network, which includes a generator and two discriminators for adversarial games; S24, the generator is used to generate a fused image, the target discriminator is used to evaluate the intensity consistency between the infrared target and the fused target, and the detail discriminator is used to distinguish the gradient distribution of the visible light target and the fused target; S25: Output the fused image for detection and use the YOLOv5 algorithm to complete target detection.

4. The method for accurately sensing human targets for surface emergency rescue according to claim 1 is characterized in that: The specific implementation of S3 includes: S31, extracting datum-DOM from region-DOM according to the POS data of the video frame, and replacing region-DOM with the datum-DOM as a new matching region of the video frame; S32. Unify the spatial resolution of datum-DOM and video frames. The ratio of datum-DOM is: Indicates scaling the size of datum-DOM, img D Indicates the size of the original datum-DOM, Scale D =gsd D / gsd F , gsd D is the spatial resolution of the original datum-DOM, gsd F is the spatial resolution of the video frame; S33, according to the similarity of the two images in the RGB color space, determine the best matching area of ​​the video frame on the datum-DOM; assuming (x L1 ,y L1 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on the datum-DOM in the RGB color space, (x L1 ,y L1 ) is obtained by the following formula: and are the intensities of the R, G, and B channels of the video frame, respectively; and are the intensities of the datum-DOM channels R, G, and B respectively; (x, y) is the pixel coordinate in the video frame, x = (1, 2, ..., N LF ), y=(1,2,…,N WF ), (△x1,△y1) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x1=(1,2,…,N LD -N LF ), △y1=(1,2,…,N WD -N WF );N LF and N WF are the length and width of the video frame, in pixels; N LD and N WD are the length and width of datum-DOM, in pixels; N F is the total number of pixels in the video frame, N F =N LF N WF ; S34, according to the similarity of the two images in the gradient amplitude space, determine the best matching area of ​​the video frame on the datum-DOM; assuming (x L2 ,y L2 ) is the pixel coordinate of the upper left corner of the best matching area of ​​the video frame on datum-DOM in the gradient amplitude space, (x L2 ,y L2 ) is obtained by the following formula: F x (x,y) and F y (x, y) is the partial derivative of the video frame in the x and y directions respectively; D x (△x2+x,△y2+y) and D y (△x2+x,△y2+y) are the first-order partial derivatives of datum-DOM in the x and y directions respectively; (x,y) are the pixel coordinates in the video frame, x=(1,2,…,N LF ), y=(1,2,…,N WF ), (△x2,△y2) is the pixel coordinate of the upper left corner of the video frame in datum-DOM, △x2=(1,2,…,N LD -N LF ), △y2=(1,2,…,N WD -N WF );N LF and N WF are the length and width of the video frame, in pixels; N LD and N WD They are the length and width of datum-DOM respectively, in pixels; S35, construct a robust block matching algorithm that considers both the color difference and texture complexity of the video frame, define a symbol TH that comprehensively represents the color difference amplitude and texture complexity of the video frame, and set its threshold to 20, (x L ,y L ) is the coordinate of the upper left corner of the best matching area obtained by the block matching algorithm, which can be calculated as: TH=∑ x,y |I F (x,y)-I D (△x2+x,△y2+y)| / N F ω L is the weight, r represents (x L1 ,y L1 ) and (x L2 ,y L2 ), N LF and N WF are the length and width of the video frame, in pixels. L ,y L ,N LF ,N WF ) extracting a block matching DOM from the datum-DOM, that is, the best matching area of ​​the video frame on the datum-DOM; S36, constructing polynomials of the video frame and the block matching DOM, and evenly dividing the gradient amplitude image of the video frame and the gradient amplitude image of the block matching DOM into n×n local areas, respectively, and each local area is represented by a second-order polynomial as follows: and are the local area intensities of the i-th column and the j-th column in the two images, i = (1, ..., n), j = (1, ..., n); X I =(x I ,y I ) T , X Π =(x Π ,y Π ) T , T represents the transpose of the matrix, (x I ,y I ) and (x Π ,y Π ) are the pixel coordinates of the local areas in the two images, T Ι and T Π are the generation time of the video frame and block matching DOM respectively; A Ι and A Π are the second-order coefficient matrices of their polynomials respectively; B I and B Π are the first-order coefficient vectors of their polynomials respectively; C Ι and C Π are the scalars of their polynomials respectively; are multiple parameters of the polynomial; S37. Construct a differential-difference polynomial and write it using the first-order derivative Taylor expansion as: f F (X I ,T I ) and f D (X Π ,T Π ) are the intensities of the corresponding local areas, X I and X Π are the pixel coordinates of the local area, T Ι and T Π are the generation time of the video frame and block matching DOM respectively; are multiple parameters of the polynomial, X Π =X I +ΔX; let d=0 and further get the equation of ΔX: A ΔX ΔX=L ΔX The second-order polynomial of S38 and ΔX is expressed as (x f ,y f ) are the coordinates of the local area in the video frame, a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5 are polynomial parameters; construct the correction equation: At=L x f and f is the column and row number of the local area in the video frame; t = (a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5) T , t is an unknown parameter vector to be solved; S39. According to the calculated optimal estimate t, the geodetic coordinates of the pixels in the video frame are obtained to complete the positioning.

5. The method for accurately sensing human targets for surface emergency rescue according to claim 1 is characterized in that: Also includes: S4. Identify the target behavior and complete the target status assessment.

6. The method for accurately sensing human targets for surface emergency rescue according to claim 5 is characterized in that: The specific implementation of S4 includes: S41. Extract each pair of features from the infrared image and the visible light image: and and S42. Given a feature map, use a weighted assembly strategy for fusion, and calculate the responses at all pixels as a weight matrix: and is the weighted matrix of thermal and visible changes of stream S after normalization by the aggregation function Agg(·); Multimodal feature fusion (MFF) uses two Agg(·) functions: sigmoid and softmax, which are applied to the score matrix and The corresponding index (i,j,k) of each element in is: Calculate the fusion feature f RGB and f Motion The formula is: S43, generate embedding vector z RGB and z Mption : and is the classification parameter, z RGB and z Motion The dimension is the number of categories. The value of each instance represents the prediction score of the corresponding category. The SoftMax function is used as the normalization function of the embedding vector to obtain the prediction result: S44. Combine RGB and Motion embedding vectors to generate z Fus , and generate predictions for the Fusion stream: With Fus =from RGB +with Motion S45, final classification loss L C is the sum of the losses on each flow: L C =L Fus +L RGB +L Motion Combined embedding vector z Fus Directly affected by RGB and z Motion To reduce the difference between the negative and positive effects of correlation, L emb : L emb =MSE(from RGB ,from Motion ) MSE(·) is the mean square error loss function. The MFF network minimizes the combined loss L, which is the sum of the classification loss and the embedding loss: L=L C +l emb The combined loss helps the MFF model strike a balance between representing RGB or optical flow input and classification loss; S46. Complete the recognition of abnormal human behavior and label each target with a status classification label.

7. A human target accurate perception method for surface emergency rescue according to claim 5 or 6, characterized in that: Also includes: S5. Dynamically update the state of the target, and detect and rescue human targets in the area in real time.

8. The method for accurately sensing human targets for surface emergency rescue according to claim 7, characterized in that: The specific implementation of S5 includes: S51, providing human target information and its position information and status information at a specific moment; S52, dynamically update the location information and status information according to the time sequence, and return it to the ground dispatcher to achieve full process supervision; S53. Ground dispatch to rescue human targets.

9. A human target precision perception system for surface emergency rescue, characterized in that: Includes a visual sensor module and an information processing module; The visual sensor module is used to capture infrared images and visible light images of video sequence images; The information processing module completes the detection, positioning and state estimation of the target according to the video sequence images taken by the visual sensor module, which is specifically achieved by executing steps S1-S4 of any one of claims 1-8.

10. The sensing system according to claim 9, characterized in that: The visual sensor adopts a dual-light fusion camera.

Images