A collision warning method based on image target detection and visual depth estimation

Abstract

The application discloses a collision warning method based on image target detection and visual depth estimation, and the method comprises the following steps: 1, obtaining a depth image and a color image from a camera; 2, obtaining detection information of a target based on the color image and tracking the target; 3, performing depth estimation and orientation estimation on the visual target to form target depth and orientation information; 4, forming a to-be-detected sequence according to the target depth and the target detection result; and 5, performing collision warning on the target according to the to-be-detected sequence. The application can detect the motion information of the target in real time, accurately judge the motion state of the moving target, give collision warning information, and thus guarantee the travel safety of the visually impaired.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multimedia information processing, and relates to computer vision, deep learning, binocular distance measurement and the like, in particular to a collision warning method based on image target detection and visual depth estimation. BACKGROUND

[0002] With the rapid development of human society, the quality of life of people has been greatly improved, especially for some people who are inconvenient to travel due to visual impairment, they walk on the road without safety guarantee. They can only rely on traditional guide dogs and guide sticks to assist them in judging whether there is danger in front. The existing technology is based on traditional methods to assist the blind, and the traditional method generally needs to spend a lot of effort and time, and the cost is high. With the rapid development of computer vision and neural network, it is possible to assist the visually impaired in traveling based on image information. SUMMARY

[0003] The application proposes a collision warning method based on image target detection and visual depth estimation to predict the collision of the visually impaired, so as to ensure the safety of travel.

[0004] The application solves the technical problems by using the following method scheme:

[0005] The collision warning method based on image target detection and visual depth estimation has the following characteristics:

[0006] Step 1, obtaining a depth image and a color image from a camera and performing preprocessing;

[0007] Step 1.1, taking the position of the color camera as the position of the user, using the color camera to obtain the color image p of the user's field of view, the format of the color image p is W p ×H p ×channel p , wherein W p , H p and channel p are the width, height and channel number of the color image p respectively; the horizontal and vertical field angles of the color camera are denoted as H rgb and V rgb ;

[0008] Step 1.2, using the depth camera to obtain the depth image d of the user's field of view, the format of the depth image d is channel d ×W d ×H d , wherein channel d , W d and Hd respectively the channel, width and height of the depth image d; let the horizontal and vertical field of view angles of the depth camera be denoted as H dist , V dist ;

[0009] Step 1.3, cropping the depth image d according to the size of the color image p to obtain two images with pixel-to-pixel matching, denoted as img color and img dist ;

[0010] Step 1.4, performing scaling and padding of gray pixels preprocessing on the color image img color to obtain a preprocessed color image with dimensions , and then performing dimension expansion operation on the preprocessed color image to obtain an expanded color image with format , wherein batch represents the expanded dimension, and then normalizing the RGB values of the pixels in the expanded color image to obtain a color input image img colorp ; wherein represents the width of the input image required by the target detection network, represents the height of the input image required by the target detection network;

[0011] Step 2, obtaining the detection information of the target based on the color image and tracking the target;

[0012] Step 2.1, inputting the color input image img colorp into the target detection network and performing processing to output the target detection information of the color input image img colorp , including: the class of all targets, the position information of the target, and the confidence information; wherein the position information of the target is a rectangular box, and the position information of the rectangular box is represented by a four-tuple composed of the upper left corner coordinates, height and width of the rectangular box;

[0013] Step 2.2, using a target tracking algorithm to detect the target detection information of the color input image img colorp to determine n+1 targets, and selecting any one target as an observation target, denoted as obs, and the remaining n targets as test targets, denoted as test1, test2,..., test i ,..., test n ; wherein test i represents the i-th test target of the color input image img colorp ;

[0014] wherein the position information of the observation target obs is recorded as a quadruple (x obs ,y obs ,w obs ,h obs ), wherein x obs ,y obs ,w obs ,h obs respectively represent the horizontal coordinate, the vertical coordinate of the upper left corner of the rectangular frame of the observation target obs, the width, the height of the rectangular frame;

[0015] Step 3, depth estimation and orientation estimation are performed on the visual target to form target depth Depth obs and orientation information;

[0016] Step 4, a sequence to be detected SDseq obs,time is formed according to the target depth and the target detection result;

[0017] Step 5, collision warning of the target is performed according to the sequence to be detected.

[0018] Step 5.1, let σ 2 be an angle variance threshold, let k stand be a collision critical slope, let a collision threshold be λ, let P be a detection upper limit, and let Sum P be the number of non-collision counts within time x P time, initialized as Sum P = 0;

[0019] Step 5.2, let p count be the detection number, initialized as p count = 1, and let count obs,time be the non-collision flag of the observation target obs within the time period, initialized as count obs,time = 0;

[0020] Step 5.3, calculate the variance var obs,time of the set of offset angles θseq obs,time of the observation target obs relative to the user position, if var obs,time < σ 2 , then directly execute step 5.4, otherwise, let count obs,time = 1 and then execute step 5.4;

[0021] Step 5.4, perform regression straight line analysis on the sequence formed by the N+1 time points corresponding to all the corrected target depths in the sequence SDseq obs,time together with the sequence SDseq obs,time , thereby obtaining the best fitting straight line, and let the slope of the best fitting straight line be k obs,time , if kobs ≥k stand Then let count obs,time If the result is 1, proceed to step 5.5; otherwise, proceed directly to step 5.5.

[0022] Step 5.5: Obtain the transformed timestamp t′ using equations (13) and (14). cap The number of transformed Sum′ p and P count The value increases by 1:

[0023]

[0024] Sum′ P =Sum P +count time,obs (14)

[0025] Step 5.6: Repeat steps 4.2-4.6 and 5.2-5.5 until P. count =P; if Sum′ p If the value is less than λ, then there is no collision; otherwise, there is a collision.

[0026] The collision warning method based on image target detection and visual depth estimation described in this invention is also characterized in that step 3 includes:

[0027] Step 3.1: Use equation (1) to analyze the location information (x) of the target obs. obs y obs w obs h obs After transformation, the position information of the target object after obs transformation is obtained.

[0028]

[0029] In equation (1), x LT This represents the x-coordinate of the top-left corner of the target object after the OBS transformation, y LT This represents the ordinate of the top-left corner of the target object after the OBS transformation, x. RB This represents the x-coordinate of the lower right corner of the target object after the OBS transformation, y RB This represents the ordinate of the lower right corner of the target object after the OBS transformation;

[0030] Step 3.2: Obtain the i-th test target test following the process in Step 3.1. i Position Quadruple in, Represents the i-th test target, test. i The transformed x-coordinate of the top left corner. Represents the i-th test target, test. i The transformed top-left ordinate. Represents the i-th test target, test. i The transformed x-coordinate of the bottom right corner. Represents the i-th test target, test. i The transformed ordinate of the lower right corner;

[0031] Step 3.3: Define the set sequence of the target obs to be examined. obj Used to save the corrected position information of the target object and initialize it. (,) represents a coordinate point in the set;

[0032] Step 3.4: Use equation (2) to obtain the i-th test target test. i Detection conditions:

[0033]

[0034] In equation (2), x min，i x max,i Represents the i-th test target, test. i The corrected x-coordinate of the target object obs, y min，i y max,i Represents the i-th test target, test. i Corrected ordinate of the target object obs;

[0035] Step 3.5, if x min，i ≤x max,i Or y min，i ≤y max,i Then keep the set sequence obj Otherwise, (x) max,i ,y max,i ), (x min，i y min，i ), (x min，i y max,i ), (x max,i ,y min，i Add to collection sequence obj middle;

[0036] Step 3.6: Following the process in Steps 3.4-3.5, complete the detection of all test targets against the observed target obs, thereby obtaining the final set sequence. obj ;

[0037] Step 3.7, from image img dist Get the final collection sequence from the source.obj The depth information of the region surrounded by all coordinate points, and the depth information of each pixel point in the depth information of the region is sorted, so as to obtain the sorted depth information of the region AreaDepth obs ;

[0038] Step 3.8, determining the estimated depth Depth of the observation target obs by using formula (4) obs :

[0039]

[0040] In formula (4), length(AreaDepth obs ) represents the number of elements in the depth information of the region AreaDepth obs ; depth j represents a certain element in the depth information of the region AreaDepth obs ;

[0041] Step 3.9, calculating the offset angle θ of the observation target obs relative to the position of the user by using formula (5) obs :

[0042]

[0043] In formula (5), X MID represents the horizontal coordinate of the center position of the rectangular frame of the observation target obs, and

[0044] The step 4 comprises:

[0045] Step 4.1, setting the number of pictures read by the color camera and the depth camera at a unit time as fps read , setting the period of the observation target obs as time, setting the time stamp as t cap , defining the current time as t Now , and initializing t cap =t Now ;

[0046] Step 4.2, obtaining 1 color image and 1 depth image at any time from t cap -time to t cap , and obtaining an estimated depth and an offset angle of the observation target obs according to the process of step 1-step 3, so as to process the color image and the depth image obtained at each time from t cap -time to t cap , and obtaining a sequence Dseq composed of N+1 estimated depths of the observation target obs obs,timeand the set θseq consisting of N+1 offset angles obs,time ;

[0047] Let t cap -time until t cap The estimated depth sequence Dseq of the target obs within a given time period obs,time The v-th estimated depth is denoted as Depth. v , let t cap -time until t cap The set of offset angles θseq of the target obs within a given time period. obs,time The v-th offset angle in the equation is denoted as θ. v ;

[0048] Step 4.3: When v = 1, calculate the (v-1)th corrected state matrix. Calculate the (v-1)th corrected target depth d v-1 =Depth v Calculate the (v-1)th corrected state variance

[0049] Step 4.4: Calculate the (v-1)th state transition matrix. Where k is the offset coefficient, Δt v-1 For the v-th estimated depth v and the v-1th estimated depth. v-1 The difference between the corresponding time intervals is used to calculate the v-th predicted state matrix. Calculate the variance of the v-th predicted value in, F represents v-1 transpose;

[0050] Step 4.5: Calculate the v-th gain matrix. Where R is the error matrix of the depth camera; calculate the v-th corrected state variance. Calculate the v-th observation state matrix Calculate the v-th corrected state matrix Then let the v-th corrected target depth d v For the v-th corrected state matrix The value in the first row and first column;

[0051] Step 4.6: After assigning v+1 to v, return to step 4.4 and execute sequentially until v > N, thus obtaining the N+1 corrected target depth sequences SDseq. obs,time ={d v |v=0,1,2,...,N}.

[0052] The electronic device comprises a memory and a processor, and is characterized in that the memory is used for storing a program supporting the processor to execute any of the collision warning methods, and the processor is configured to execute the program stored in the memory.

[0053] The computer readable storage medium stores a computer program, and the computer program is characterized in that when the computer program is run by a processor, the steps of any of the collision warning methods are executed.

[0054] Compared with the prior art, the beneficial effects of the present application are reflected in that:

[0055] 1、The present application uses deep learning technology to process the content captured by the camera, and proposes a collision warning method based on image depth information and target detection information, which can realize continuous video or camera capture picture analysis, identify the target in the picture and track the position of each target, and refine the position of each target, and according to the depth information, the collision of pedestrians and objects can be warned, and the collision result is given, so as to adjust the walking direction of the pedestrians, and ensure the safety of the visually impaired and ordinary pedestrians.

[0056] 2、The present application uses the target detection network to process the distance information and direction information in the movement of the object, reduces the calculation and storage consumption, improves the operation speed, and significantly enhances the stability and accuracy of the collision warning.

[0057] 3、The present application can process the distance information and direction information in the movement of the object, and improve the stability and accuracy of the direction judgment.

[0058] 4、The collision warning method can effectively distinguish the sudden situation encountered by the visually impaired person during walking, such as the sudden approach of the object or the collision risk, and judge the object blocking in the walking process of the pedestrian, fully use the depth information of the image, get more accurate target depth, and prompt the pedestrian, improve the accuracy of the judgment result, and thus ensure the safety of the visually impaired person. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 It is a schematic diagram of the collision warning method in the embodiment of the present application.

[0060] Figure 2 It is a target detection flowchart in the present application.

[0061] Figure 3 It is an object position information correction algorithm flowchart in the present application.

[0062] Figure 4Flowchart for collision calculation in the present application. DETAILED DESCRIPTION

[0063] In this embodiment, a collision warning method based on target detection and visual depth estimation is used. The method uses computer vision, deep learning, binocular distance measurement and other technologies to measure the distance of the identified target, determine the direction of the target, and finally give a collision warning for the on-site environment and real-time feedback and monitoring of the environment information for the visually impaired. As shown in Figure 1 , the method is performed as follows:

[0064] Step 1, obtain depth image and color image from camera and pre-process;

[0065] Step 1.1, take the position of the color camera as the position of the user, use the color camera to obtain the color image p of the user's field of view, the format of the color image p is W p ×H p ×channel p , where W p , H p and channel p are the width, height and channel number of the color image p respectively; the horizontal and vertical field of view angles of the color camera are denoted as H rgb , V rgb ;

[0066] Step 1.2, use the depth camera to obtain the depth image d of the user's field of view, the format of the depth image d is channel d ×W d ×H d , where channel d , W d and H d are the channel, width and height of the depth image d respectively; let the horizontal and vertical field of view angles of the depth camera be denoted as H dist , V dist ;

[0067] Step 1.3, according to the size of the color image p, crop the depth image d to obtain two images with pixel-to-pixel matching, denoted as img color and img dist ;

[0068] Step 1.4, perform scaling and padding of gray pixels on the color image img color , to obtain a pre-processed color image with dimensions , and then perform dimension expansion on the pre-processed color image to obtain a color image with format extended color image, wherein, batch represents the dimension of the extended color image, and RGB values of pixels in the extended color image are normalized to obtain a color input image img colorp ; wherein, represents the width of the input image required by the target detection network, represents the height of the input image required by the target detection network;

[0069] Step 2, obtaining the detection information of the target based on the color image and tracking the target;

[0070] Step 2.1, inputting the color input image img colorp into the target detection network and processing, and outputting the target detection information of the color input image img colorp , including: the category of all targets, the position information of the target, and the confidence information; wherein, the position information of the target is a rectangular frame, and the position information of the rectangular frame is represented by a four-tuple composed of the left upper corner coordinates, height and width of the rectangular frame, as shown in Figure 2 , the target detection network is an open source YOLO target detection network, which is realized after optimization;

[0071] Step 2.4, using a target tracking algorithm to detect the target detection information of the color input image img colorp , so as to determine n+1 targets, and select any one target as an observation target, denoted as obs, and the remaining n targets as test targets, denoted as test1, test2,..., test i ,..., test n ; wherein, test i represents the i-th test target of the color input image img colorp ;

[0072] wherein, let the position information of the observation target obs be a four-tuple (x obs , y obs , w obs , h obs ), wherein, x obs , y obs , w obs , h obs represent the horizontal coordinate, vertical coordinate of the left upper corner of the rectangular frame of the observation target obs, and the width and height of the rectangular frame, respectively;

[0073] Step 3, depth estimation and orientation estimation of the visual target are performed to form target depth and orientation information;

[0074] Step 3.1, since the pedestrians on the road can appear to be occluded, resulting in a decrease in the accuracy of the depth processing, the depth needs to be finely processed to improve the accuracy, and the position information (x obs , y obs , w obs , h obs ) of the observation target obs is transformed using equation (1) to obtain the transformed position information

[0075]

[0076] In equation (1), x LT represents the horizontal coordinate of the upper left corner of the transformed observation target obs, y LT represents the vertical coordinate of the upper left corner of the transformed observation target obs, x RB represents the horizontal coordinate of the lower right corner of the transformed observation target obs, and y RB represents the vertical coordinate of the lower right corner of the transformed observation target obs.

[0077] Step 3.2, the position quadruple of the ith test target test i is obtained according to the process of step 3.1 wherein, represents the horizontal coordinate of the upper left corner of the transformed ith test target test i , represents the vertical coordinate of the upper left corner of the transformed ith test target test i , represents the horizontal coordinate of the lower right corner of the transformed ith test target test i , represents the vertical coordinate of the lower right corner of the transformed ith test target test i .

[0078] Step 3.3, define a set sequence obj of observation targets obs for saving the corrected position information of the observation targets obs, and initialize (, ) represents a coordinate point in the set.

[0079] Step 3.4, the detection condition of the ith test target test i is obtained using equation (2):

[0080]

[0081] In equation (2), x min，i , x max,i represents the horizontal coordinate of the upper left corner of the transformed ith test target test iThe corrected x-coordinate of the target object obs, y min，i y max,i Represents the i-th test target, test. i Corrected ordinate of the target object obs;

[0082] Step 3.5, if x min，i ≤x max,i Or y min，i ≤y max,i Then keep the set sequence obj Otherwise, (x) max,i y max,i ), (x min，i y min，i ), (x min，i y max,i ), (x max,i y min，i Add to collection sequence obj middle;

[0083] Step 3.6: Following the process in Steps 3.4-3.5, complete the detection of all test targets against the observed target obs, thereby obtaining the final set sequence. obj ;

[0084] Step 3.7, from image img dist Get the final collection sequence from the source. obj The depth information of the region enclosed by all coordinate points is obtained, and the depth information of each pixel in the region is sorted to obtain the sorted depth information AreaDepth. obs ;

[0085] Step 3.8: Use equation (4) to determine the estimated depth of the target object obs. obs :

[0086]

[0087] In equation (4), length(AreaDepth) obs AreaDepth represents the depth information of the region. obs The number of elements in the middle; depth j AreaDepth represents the depth information of the region. obs For a given element, the entire algorithm flow is as follows: Figure 3 As shown;

[0088] Step 3.9: Calculate the offset angle θ of the target obs relative to the user's position using equation (5). obs :

[0089]

[0090] In formula (5), X MID represents the horizontal coordinate of the center position of the rectangular frame of the observation target obs, and

[0091] Step 4, forming a to-be-detected sequence according to the target depth and the target detection result;

[0092] Step 4.1, since the camera may shake during the collection of images, the shaking will greatly affect the accuracy of data collection, and therefore the depth collected under the shaking of the camera needs to be processed. The number of pictures read by the color camera and the depth camera at a unit time is set to fps read , the period of the observation target obs is set to time, the timestamp is set to t cap , the current time is defined as t Now , and t cap = t Now is initialized;

[0093] Step 4.2, at any time from t cap -time to t cap , one color image and one depth image are acquired, and an estimated depth and an offset angle of the observation target obs are obtained according to the process of steps 1-3, so that the color image and the depth image acquired at each time from t cap -time to t cap are processed, and a sequence Dseq obs,time consisting of N+1 estimated depths of the observation target obs and a set θseq obs,time consisting of N+1 offset angles are obtained;

[0094] Let the vth estimated depth in the estimated depth sequence Dseq cap of the observation target obs from t cap -time to t obs,time be Depth v , and let the vth offset angle in the offset angle set θseq cap of the observation target obs from t cap -time to t obs,time be θ v ;

[0095] Step 4.3, when v=1, the v-1th correction state matrix is calculated, and the v-1th corrected target depth d v-1 = Depth v, the v-1th modified state variance is calculated

[0096] Step 4.4, the v-1th state transition matrix is calculated wherein k is the offset coefficient, Δt v-1 is the vth estimated depth Depth v and the v-1th estimated depth Depth v-1 corresponding to the time difference, the vth predicted state matrix is calculated the vth predicted value variance is calculated wherein, represents the transpose of F v-1 ;

[0097] Step 4.5, the vth gain matrix is calculated wherein R is the error matrix of the depth camera; the vth modified state variance is calculated the vth observed state matrix is calculated the vth modified state matrix is calculated then the vth modified target depth d v is the vth modified state matrix the value of the first row and the first column;

[0098] Step 4.6, after v is assigned to v+1, the step 4.4 is executed in sequence until v>N, thereby obtaining a sequence SDseq composed of N+1 modified target depths obs,time ={d v |v=0, 1, 2, …, N};

[0099] Step 5, the collision warning of the target is performed according to the to-be-detected sequence.

[0100] Step 5.1, let σ 2 be the angle variance threshold, let k stand be the collision critical slope, let the collision threshold be λ, let P be the detection upper limit, and let Sum P be the number of non-collision counts being 1 within the time×P time, initialized as Sum P =0;

[0101] Step 5.2, let p count be the detection number, initialized as p count =1, and let count obs,time be the non-collision flag of the target obs within the time period, initialized as count obs,time =0;

[0102] Step 5.3, the offset angle set θseq of the target obs relative to the user position is calculatedobs,time the variance var obs,time if var obs,time < σ 2 , then directly execute step 5.4, otherwise, let count obs,time = 1 and then execute step 5.4;

[0103] Step 5.4, perform regression straight line analysis on the sequence composed of N+1 time points corresponding to all the modified target depths in the sequence SDseq obs,time and the sequence SDseq obs,time , thereby obtaining the best fitting straight line, and let the slope of the best fitting straight line be k obs,time if k obs ≥ k stand , then let count obs,time = 1 and then execute step 5.5; otherwise, directly execute step 5.5

[0104] Step 5.5, obtain the transformed timestamp t' and the transformed number Sum' by using formula (13) and formula (14), and increase the value of P by 1: cap p count

[0105]

[0106] Sum' P = Sum P + count time,obs (14)

[0107] Step 5.6, repeat steps 4.2-4.6 and steps 5.2-5.5 until P count = P; if Sum' p < λ, it is determined that there is no collision, otherwise, it is determined that there is a collision, and the flow of this step is shown in Figure 4

[0108] In this embodiment, an electronic device includes a memory and a processor, the memory is used to store a program supporting the processor to execute the above-mentioned collision warning method, and the processor is configured to execute the program stored in the memory.

[0109] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above-mentioned collision warning method.

[0110] ​​​​In summary, the collision warning method based on image depth information and target detection information solves the occlusion problem of the target on the road, solves the difficulty and safety problem of the visually impaired person in outdoor travel, and guarantees the travel safety of the visually impaired person.

Claims

1. A collision warning method based on image target detection and visual depth estimation, characterized in that, is performed according to the following steps: Step 1, obtaining a depth image and a color image from a camera and pre-processing; Step 1.1: Using the position of the color camera as the user's position, obtain a color image p of the user's field of vision using the color camera. The format of the color image p is W. p ×H p ×channel p Among them, W p H p and channel p These represent the width, height, and number of channels of the color image p, respectively; the horizontal and vertical field of view of the color camera are denoted as H. rgb V rgb ; Step 1.

2. Obtain a depth image d of the user's field of view using a depth camera, the depth image d being in the format channel d x W d x H d , where channel d , W d , and H d are the channel, width, and height of the depth image d, respectively; let the horizontal and vertical field of view angles of the depth camera be denoted by H dist , V dist ; Step 1.

3. Crop the depth image d according to the size of the color image p, to get two images with pixel-to-pixel matching, denoted as img color and img dist ; Step 1.4, the color image img color The preprocessing of scaling and filling gray pixels is performed to obtain a preprocessed color image with dimensions of A dimension expansion operation is performed on the preprocessed color image to obtain an expanded color image with a format of , wherein batch represents the expanded dimensions, and the RGB values of the pixel points in the expanded color image are normalized to obtain a color input image img colorp ; wherein represents the width of the input image required by the target detection network, represents the height of the input image required by the target detection network. Step 2, obtaining detection information of a target based on the color image and tracking the target; Step 2.1, inputting a color image img colorp After the target detection network is inputted and processed, target detection information of the color image img colorp is outputted, including: class of all targets, position information of the targets, and confidence information; wherein the position information of the targets is a rectangular frame, and the position information of the rectangular frame is represented by a four-tuple composed of the upper left corner coordinates, height and width of the rectangular frame. Step 2.

2. Detecting target detection information of the color input image img colorp by using a target tracking algorithm, so as to determine n+1 targets, and select any one target as an observation target, recorded as obs, and the remaining n targets as test targets, recorded as test1, test2, …, test i n. n ; wherein test i i represents the i-th test target of the color input image img colorp . wherein the position information of the observation target obs is recorded as a four-tuple (x obs ,y obs ,w obs ,h obs ), wherein x obs ,y obs ,w obs ,h obs respectively represent the horizontal coordinate, the vertical coordinate of the upper left corner of the rectangular frame of the observation target obs, the width, the height of the rectangular frame. Step 3, depth and orientation estimation of visual targets, forming target depth Depth obs and orientation information; Step 4, forming a sequence SDseq to be detected according to the target depth and the target detection result obs,time ; Step 5, collision warning of the target according to a to-be-detected sequence; Step 5.1, let σ 2 be the angle variance threshold, let k stand be the collision critical slope, let the collision threshold be λ, let P be the detection upper limit, let Sum P be the number of non-collision counts in time x P time, initialized to Sum P = 0; Step 5.2, let p count = 1, let count count = 0; for each time in the time period obs,time = 0; for each time in the time period obs,time = 0; for each time in the time period Step 5.

3. Calculate a set of offset angles θseq of the observation target obs with respect to the user position obs,time variance var of the angles θseq obs,time If var is satisfied, directly execute Step 5.4, otherwise, let count = 1 and execute Step 5.4 after count obs,time <σ 2 obs,time = 1;​ Step 5.4, the sequence SDseq obs,time corresponding to all the modified target depths is subjected to regression straight line analysis together with the sequence SDseq obs,time , and the best fitting straight line is obtained, with the slope of the best fitting straight line being k obs,time If k obs ≥ k stand , then count obs,time = 1 and step 5.5 is executed; otherwise, step 5.5 is directly executed Step 5.5, obtain transformed time stamp t' using formula (13) and formula (14) cap and the number of transformations Sum' p and increase the value of P count by 1 : Sum P = Sum P + count time,obs (14) Step 5.6, repeat steps 4.2-step 4.6, steps 5.2-step 5.5 until P count = P; if Sum' p < λ, then determine no collision, otherwise, determine collision.

2. The collision warning method based on image target detection and visual depth estimation according to claim 1, characterized in that, the step 3 comprises: Step 3.1, transform the position information (x obs ,y obs ,w obs ,h obs ) of the observation target obs using formula (1) to obtain the transformed position information of the observation target obs In formula (1), x LT represents the horizontal coordinate of the upper left corner of the observation target obs after transformation, y LT represents the vertical coordinate of the upper left corner of the observation target obs after transformation, x RB represents the horizontal coordinate of the lower right corner of the observation target obs after transformation, y RB represents the vertical coordinate of the lower right corner of the observation target obs after transformation; Step 3.

2. Obtain the position quad of the ith test target test i i according to the procedure of step 3.1 wherein, represents the position quad of the ith test target test i the horizontal coordinate of the transformed upper left corner, represents the position quad of the ith test target test i the vertical coordinate of the transformed upper left corner, represents the position quad of the ith test target test i the horizontal coordinate of the transformed lower right corner, represents the position quad of the ith test target test i the vertical coordinate of the transformed lower right corner; Step 3.3, define the set of observation targets obs sequence obj for saving the modified position information of the observation target obs, and initializing (,) represents a coordinate point in the set; Step 3.4, obtaining the ith test target test i of detection conditions: In formula (2), x min,i , x max,i represents the modified abscissa of the i-th test target test i obs; y min,i , y max,i represents the modified ordinate of the i-th test target test i obs; Step 3.5, if x min,i ≤ x max,i or y min,i ≤ y max,i , then keep the set sequence obj unchanged, otherwise, add (x max,i , y max,i ), (x min,i , y min,i ), (x min,i , y max,i ), (x max,i , y min,i ) to the set sequence obj ; Step 3.

6. Complete the detection of all test targets on the investigation target obs according to the process of Step 3.4-Step 3.5, so as to obtain the final set sequence obj ; Step 3.7, obtaining the final set sequence from the image img dist obj obs ;​​ Step 3.8, determining an estimated depth, Depth, of the object of interest, obs, using the formula (4) obs : In formula (4), length(AreaDepth obs ) represents the depth information AreaDepth obs of the region j ; and depth obs represents a certain element in the depth information AreaDepth obs of the region. Step 3.9, Calculate the offset angle Θ of the target obs with respect to the user's position using formula (5) obs : In formula (5), X MID represents the horizontal coordinate of the center position of the rectangular frame of the observation target obs, and 3.The collision warning method based on image target detection and visual depth estimation of claim 2, wherein, the step 4 comprises: Step 4.1, let the number of pictures read by the color camera and the depth camera at a unit time be fps read , set the period of the observation target obs to be time, and set the time stamp to be t cap , define the current time to be t Now , and initialize t cap = t Now ; Step 4.2, at t cap -time until t cap At any given time, acquire one color image and one depth image, and obtain an estimated depth and offset angle of the target object (obs) by following steps 1-3, thereby determining the depth of the target object (obs) at time t. cap -time until t cap The color images and depth maps acquired at each time step are processed to obtain a sequence Dseq consisting of N+1 estimated depths of the target object obs. obs,time and the set θseq consisting of N+1 offset angles obs,time ; Let t cap -time until t cap The estimated depth sequence Dseq of the target obs within a given time period obs,time The v-th estimated depth is denoted as Depth. v , let t cap -time until t cap The set of offset angles θseq of the target obs within a given time period. obs,time The v-th offset angle in the equation is denoted as θ. v ; Step 4.

3. When v = 1, calculate the v - 1th modified state matrix Calculate the v - 1th modified target depth d v-1 = Depth v , calculate the v - 1th modified state variance Step 4.4: Calculate the (v-1)th state transition matrix. Where k is the offset coefficient, Δt v-1 For the v-th estimated depth v and the v-1th estimated depth. v-1 The difference between the corresponding time intervals is used to calculate the v-th predicted state matrix. Calculate the variance of the v-th predicted value in, F represents v-1 transpose; Step 4.5, compute the vth gain matrix where R is the error matrix of the depth camera; compute the vth modified state variance compute the vth observation state matrix compute the vth modified state matrix then let the vth modified target depth d v be the vth modified state matrix the value in the first row first column; Step 4.6, after v+1 is assigned to v, return to step 4.4 to sequentially execute until v>N, thereby obtaining a sequence SDseq of N+1 modified target depths obs,time = {d v |v=0,1,2,…,N}.

4. An electronic device comprising a memory and a processor, characterized in that the memory is used for storing a program supporting the processor to execute the collision warning method in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that the computer program is executed by the processor to perform the steps of the collision warning method in any one of claims 1-3.

Images