Obstacle detection method and device, storage medium and electronic device

Abstract

The application discloses an obstacle detection method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring an image to be detected which is captured by a mobile terminal; inputting the image to be detected into an obstacle detection model to obtain an obstacle recognition result of the image to be detected, wherein the obstacle recognition result is used to represent the area of the target type of the recognized obstacle in the image to be detected; inputting the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected, wherein the depth estimation result is used to represent the distance between the object represented by each pixel point and the mobile terminal; and determining the distance between the target type of the obstacle and the mobile terminal according to the obstacle recognition result and the depth estimation result. The application solves the technical problem of high cost caused by the dependence of obstacle detection on multiple sensors.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a method and apparatus for detecting obstacles, a storage medium, and an electronic device. Background Technology

[0002] Obstacle detection can be applied to various technologies, such as robot navigation systems and autonomous driving systems. Currently, it requires combining data from multiple sensors to achieve obstacle detection, such as radar, infrared sensors, and ultrasonic sensors.

[0003] In one application scenario, multiple sensors mounted on a wearable device can detect obstacles and alert blind users to these obstacles. However, the high cost of multiple sensors in wearable devices increases their complexity and limits their user base.

[0004] There is currently no effective solution to the problem of high costs caused by the reliance on multiple sensors in obstacle detection technologies. Summary of the Invention

[0005] This invention provides an obstacle detection method and apparatus, storage medium and electronic device, to at least solve the technical problem of high cost caused by the need for multiple sensors in obstacle detection.

[0006] According to one aspect of the present invention, an obstacle detection method is provided, comprising: acquiring an image to be detected captured by a mobile terminal; inputting the image to be detected into an obstacle detection model to obtain an obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify an obstacle of a target type in the image to be detected, and the obstacle recognition result is used to represent the region of the identified obstacle of the target type in the image to be detected; inputting the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected, wherein the depth estimation model is used to perform depth estimation on each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal; and determining the distance between the obstacle of the target type and the mobile terminal based on the obstacle recognition result and the depth estimation result.

[0007] Optionally, the step of inputting the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected includes: performing N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected in an iterative manner through the depth estimation model to obtain a target downsampling result, wherein N≥2 and N is a natural number, each set of downsampling operations includes a convolution operation and a pooling operation, and the convolution operation in each set of downsampling operations is performed to obtain a corresponding convolution result; performing N sets of upsampling operations on the target downsampling result in an iterative manner through the depth estimation model to obtain a target upsampling result, wherein each set of upsampling operations includes a convolution operation, a deconvolution operation and a feature fusion operation, and the feature fusion operation in each set of upsampling operations is used to fuse the deconvolution result obtained by the deconvolution operation and the convolution result obtained in the corresponding set of downsampling operations; and determining the depth estimation result of the image to be detected based on the target upsampling result.

[0008] Optionally, the step of iteratively performing N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected using the depth estimation model to obtain the target downsampling result includes: performing the i-th set of downsampling operations through the following steps, wherein the initial value of the previous downsampling result is the pixel value of the pixel in the image to be detected, 1≤i≤N, and the target downsampling result is the N-th downsampling result: performing multiple convolution operations on the previous downsampling result using the i-th convolution kernel in an iterative manner to obtain the i-th convolution result; performing the pooling operation in the i-th set of downsampling operations on the i-th convolution result to obtain the i-th downsampling result.

[0009] Optionally, the previous downsampling result includes A i-1 ×B i-1 ×C i-1 Feature data, where A i-1 B represents the height of the image obtained after the last downsampling operation. i-1 C represents the width of the image obtained after performing the previous downsampling operation. i-1 This represents the number of channels in the image obtained by performing the previous downsampling operation; the step of performing multiple convolution operations on the previous downsampling result using the i-th convolution kernel in an iterative manner to obtain the i-th convolution result includes: using the i-th convolution kernel to perform multiple convolution operations on the image obtained by the previous downsampling operation. i-1 ×B i-1 ×C i-1 The feature data is subjected to Q convolution operations in an iterative manner to obtain the i-th convolution result, wherein the i-th convolution result includes (A i-1 -Q×M i )×(B i-1 -Q×M i)×C i-1 Feature data, M i Corresponding to the size of the i-th convolutional kernel; the step of performing the pooling operation in the i-th group of downsampling operations on the i-th convolutional result to obtain the i-th downsampling result includes: for the (A i-1 -Q×M i )×(B i-1 -Q×M i )×C i-1 The feature data is subjected to pooling operation in the i-th group of downsampling operations to obtain the i-th downsampling result, wherein the i-th downsampling result includes A i ×B i ×C i Feature data, A i =(A i-1 -Q×M i ) / P i B i = (B i-1 -Q×M i ) / P i C i =C i-1 ×P i P i It corresponds to the size of the pooling window used in the pooling operation of the i-th group of downsampling operations.

[0010] Optionally, the step of iteratively performing N sets of upsampling operations on the target downsampling result using the depth estimation model to obtain the target upsampling result includes: performing the j-th set of upsampling operations through the following steps, wherein the initial value of the previous upsampling result is the target downsampling result, 1≤j≤N, and the target upsampling result is the N-th upsampling result: performing multiple convolution operations on the previous upsampling result using the j-th convolution kernel in an iterative manner to obtain the j-th convolution result; performing the deconvolution operation in the j-th set of upsampling operations on the j-th convolution result to obtain the j-th deconvolution result; and performing feature fusion on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th set of downsampling operations to obtain the j-th upsampling result.

[0011] Optionally, the previous upsampling result includes A j-1 ×B j-1 ×C j-1 Feature data, where A j-1 B represents the height of the image obtained after the last upsampling operation. j-1 C represents the width of the image obtained after performing the previous upsampling operation. j-1This represents the number of channels in the image obtained by performing the previous upsampling operation; the step of performing multiple convolution operations on the previous upsampling result using the j-th convolution kernel in an iterative manner to obtain the j-th convolution result includes: using the j-th convolution kernel to perform multiple convolution operations on the image obtained by the previous upsampling operation. j-1 ×B j-1 ×C j-1 The feature data is subjected to H convolution operations in an iterative manner to obtain the j-th convolution result, wherein the j-th convolution result includes (A j-1 -H×M j )×(B j-1 -H×M j )×(C j-1 -H×T j Feature data of M j and T j Corresponding to the size of the j-th convolution kernel; the step of performing the deconvolution operation in the j-th group of upsampling operations on the j-th convolution result to obtain the j-th deconvolution result includes: performing the deconvolution operation on the (A) kernel. j-1 -H×M j )×(B j-1 -H×M j )×(C j-1 -H×T j The feature data of ) undergoes a deconvolution operation in the j-th upsampling operation to obtain the j-th deconvolution result, wherein the j-th upsampling result includes D j ×E j ×F j Feature data, D j =(A j-1 -H×M j )×P j E j = (B j-1 -H×M j )×P j F j =C j-1 -H×T j P j It corresponds to the size of the third convolution kernel used in the deconvolution operation of the j-th group of upsampling operations.

[0012] Optionally, feature fusion is performed on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th downsampling operation to obtain the j-th upsampling result, including: selecting D from the convolution results obtained in the (N-j+1)-th downsampling operation. j ×E j ×F jThe feature data is used to obtain the j-th group of data to be fused; the j-th group of data to be fused and the j-th deconvolution result are fused to obtain the j-th upsampling result, wherein the j-th upsampling result includes A j ×B j ×C j Feature data.

[0013] Optionally, determining the depth estimation result of the image to be detected based on the target upsampling result includes: inputting the target upsampling result into a fully connected layer in the depth estimation model, and obtaining a distance estimate corresponding to the pixel value of each pixel in the target upsampling result through the fully connected layer. The depth estimation model is a model trained using a set of sample data on an original neural network model. The set of sample data includes sample images and a known distance between the object represented by each pixel in the sample images and the sample terminal. The sample images are images captured by the sample terminal. The distance estimate of the sample images output by the fully connected layer in the trained depth estimation model satisfies a preset convergence condition with the known distance. The height of the image in the target upsampling result is the same as the height of the image to be detected, and the width of the image in the target upsampling result is the same as the width of the image to be detected.

[0014] Optionally, the image to be detected is input into an obstacle detection model to obtain an obstacle recognition result for the image to be detected, including: recognizing the object to be identified in the image to be detected to obtain multiple detection boxes, wherein the detection boxes are used to represent the region of the object to be identified in the image to be detected; determining a target detection box from the multiple detection boxes, wherein the object to be identified within the target detection box is an obstacle of the target type; and outputting the region of the target detection box in the image to be detected, and the target type.

[0015] Optionally, determining the distance between the obstacle of the target type and the mobile terminal based on the obstacle recognition result and the depth estimation result includes: matching the obstacle recognition result with the depth estimation result, determining a set of depth estimation values ​​from the depth estimation result, wherein the set of depth estimation values ​​is used to represent the distance between the object corresponding to each pixel of the obstacle of the target type in the image to be detected and the mobile terminal; and determining the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimation values.

[0016] Optionally, determining the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimates includes: determining the smallest depth estimate in the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal; or determining the mean of the depth estimates in the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal.

[0017] Optionally, the method further includes: displaying on the mobile terminal the detection area of ​​the obstacle of the target type and the distance between the obstacle of the target type and the mobile terminal; and / or broadcasting on the mobile terminal the orientation of the obstacle of the target type relative to the mobile terminal and the distance between the obstacle of the target type and the mobile terminal.

[0018] According to another aspect of the present invention, an obstacle detection device is also provided, comprising: an acquisition module for acquiring an image to be detected captured by a mobile terminal; a first input module for inputting the image to be detected into an obstacle detection model to obtain an obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify an obstacle of a target type in the image to be detected, and the obstacle recognition result is used to represent the region of the identified obstacle of the target type in the image to be detected; a second input module for inputting the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected, wherein the depth estimation model is used to perform depth estimation for each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal; and a determination module for determining the distance between the obstacle of the target type and the mobile terminal based on the obstacle recognition result and the depth estimation result.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described obstacle detection method at runtime.

[0020] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the obstacle detection method described above through the computer program.

[0021] In this embodiment of the invention, depth estimation and obstacle recognition are performed on the image to be detected captured by a mobile device. An obstacle detection model is used to identify the target type of obstacle in the image to be detected, and a depth estimation model is used to estimate the distance between the object represented by each pixel in the image to be detected and the mobile terminal. By combining the obstacle recognition result and the depth estimation result, the distance between the obstacle and the mobile terminal can be obtained. This achieves the goal of detecting obstacles using only the image to be detected captured by the mobile terminal, thereby achieving the technical effect of saving costs and solving the technical problem of high cost caused by the need for multiple sensors in obstacle detection. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is a schematic diagram of the application environment of an optional obstacle detection method according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic flowchart of an optional obstacle detection method according to an embodiment of the present invention;

[0025] Figure 3 This is an optional interface illustration according to an embodiment of the present invention. Figure 1 ;

[0026] Figure 4 This is an optional interface illustration according to an embodiment of the present invention. Figure 2 ;

[0027] Figure 5 This is a schematic diagram of an optional model structure according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of an optional downsampling operation structure according to an embodiment of the present invention. Figure 1 ;

[0029] Figure 7 This is a schematic diagram of an optional downsampling operation structure according to an embodiment of the present invention. Figure 2 ;

[0030] Figure 8 This is a schematic diagram of an optional upsampling operation structure according to an embodiment of the present invention;

[0031] Figure 9 This is an optional depth diagram according to an embodiment of the present invention;

[0032] Figure 10 This is an optional schematic diagram of an image to be detected according to an embodiment of the present invention;

[0033] Figure 11 This is a schematic diagram of an optional overall model structure according to an embodiment of the present invention;

[0034] Figure 12 This is an optional interface illustration according to an embodiment of the present invention. Figure 3 ;

[0035] Figure 13 This is a schematic diagram of the structure of an optional obstacle detection device according to an embodiment of the present invention;

[0036] Figure 14 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] According to one aspect of the present invention, an obstacle detection method is provided. Optionally, as an alternative implementation, the obstacle detection method may be applied to, but is not limited to, [examples of other methods]. Figure 1 The system environment shown above includes, but is not limited to, user device 102, network 110, and server 112.

[0040] Optionally, in this embodiment, the user equipment 102 may be a mobile terminal with a shooting function, including but not limited to at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, camera, etc. The user equipment 102 includes a display 108, a processor 106, and a memory 104. The display can be used to display the image to be detected captured by the mobile terminal, the processor is used to process the image to be detected, and the memory is used to store data, including but not limited to image data. The network 110 may include but not limited to: wired network and wireless network. The wired network includes: local area network, metropolitan area network, and wide area network. The wireless network includes: Bluetooth, WIFI, and other networks that enable wireless communication. The server 112 may be a single server, a server cluster composed of multiple servers, or a cloud server. The server includes a database 114 and a processing engine 115. The database is used to store data, including but not limited to image data, model data in the obstacle detection model, and model data in the depth estimation model. The processing engine is used to process the image to be detected. The above is merely an example, and no limitation is made in this embodiment.

[0041] Alternatively, as an alternative implementation method, such as Figure 2 As shown, the method for detecting the above-mentioned obstacles includes:

[0042] Step S202: Obtain the image to be detected captured by the mobile terminal;

[0043] Among them, the mobile terminal can be an electronic device such as a mobile phone or tablet computer with a camera function.

[0044] Step S204: Input the image to be detected into the obstacle detection model to obtain the obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify the target type of obstacle in the image to be detected, and the obstacle recognition result is used to represent the area of ​​the identified target type obstacle in the image to be detected;

[0045] The obstacle detection model can be a neural network model. This model can be obtained through iterative training using training data. A well-trained obstacle detection model can identify objects of corresponding types in an image, such as tasks, animals, and vehicles, to recognize obstacles and their types.

[0046] Step S206: Input the image to be detected into the depth estimation model to obtain the depth estimation result of the image to be detected. The depth estimation model is used to perform depth estimation on each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal.

[0047] The depth estimation model can be a convolutional neural network (CNN). This model can perform calculations such as convolution, pooling, and deconvolution. It can be obtained by repeatedly training the CNN with training data. The depth estimation model can estimate the distance between the object being photographed and the location of the mobile terminal when the object was photographed. By performing depth estimation on each pixel in the image to be detected, the distance between the object corresponding to that pixel and the location of the mobile terminal when the image was photographed can be estimated.

[0048] Step S208: Based on the obstacle identification result and the depth estimation result, determine the distance between the target type obstacle and the mobile terminal.

[0049] The obstacle recognition result includes the region of the obstacle in the image to be detected, and the depth estimation result includes the distance between the object corresponding to each pixel in the image to be detected and the mobile terminal. By matching the obstacle recognition result and the depth estimation result, the distance between the obstacle and the mobile terminal can be obtained.

[0050] As an optional implementation, the obstacle detection method described above can be a function of an application installed on a mobile device. The user can open the application via voice; for example, by saying "Open Xiao A," the user can open the application, which can then capture images in real time. The obstacle detection method can be a function of the mobile device, such as a mode of mobile phone photography. Figure 3 The diagram shown is an interface illustration according to an optional embodiment of the present invention. Figure 1 , Figure 3 The image shows the camera function of the mobile terminal. After entering the camera mode, users can select the blind mode. Users can activate the blind mode via voice commands. For example, a user can say "Hey, Xiao B, turn on the blind mode for taking photos" to enter the blind shooting mode. Once in blind mode, the mobile terminal can take a picture.

[0051] As an optional implementation, such as Figure 4 The diagram shown is an interface illustration according to an optional embodiment of the present invention. Figure 2The mobile terminal interface can display images captured by the mobile terminal in real time. A depth map is obtained through a depth estimation model, and the pixel value of each pixel in the depth map corresponds to the distance between the object at that pixel and the location of the mobile terminal when the image was captured. An obstacle detection model can identify obstacles in the image, and the obstacle type, location, and distance between the obstacle and the mobile terminal can be marked on the depth map. In this embodiment, the obstacle type and location can also be announced via voice, thereby enabling navigation for the blind.

[0052] Optionally, the step of inputting the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected includes: performing N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected in an iterative manner through the depth estimation model to obtain a target downsampling result, wherein N≥2 and N is a natural number, each set of downsampling operations includes a convolution operation and a pooling operation, and the convolution operation in each set of downsampling operations is performed to obtain a corresponding convolution result; performing N sets of upsampling operations on the target downsampling result in an iterative manner through the depth estimation model to obtain a target upsampling result, wherein each set of upsampling operations includes a convolution operation, a deconvolution operation and a feature fusion operation, and the feature fusion operation in each set of upsampling operations is used to fuse the deconvolution result obtained by the deconvolution operation and the convolution result obtained in the corresponding set of downsampling operations; and determining the depth estimation result of the image to be detected based on the target upsampling result.

[0053] As an alternative implementation method, for depth estimation models, such as Figure 5 The diagram illustrates an optional model structure according to the present invention. The input to this model structure is the image to be detected, and the target downsampling result is obtained by performing N consecutive downsampling operations. In this embodiment, the downsampling operation includes convolution and pooling operations, and each group of downsampling operations includes both convolution and pooling operations, such as... Figure 5 The input image is first subjected to two convolution operations followed by one pooling operation. In other words, a set of downsampling operations is performed on the detection image, including two convolution operations and one pooling operation. The number of convolutions and pooling operations in a set of downsampling operations can be determined according to the actual situation. Figure 5 The two convolutions and one pooling operation included in a set of downsampling operations are only for illustrative purposes and are not intended to be limiting. For example, a set of downsampling operations may also include three convolutions and two pooling operations.

[0054] The depth estimation model obtains a target upsampling result by performing N consecutive upsampling operations on the downsampled target result. Based on the upsampling result, the distance between the object corresponding to each pixel in the image to be detected and the location of the mobile terminal when the image was captured can be obtained. In this embodiment, the upsampling operation includes convolution, deconvolution, and feature fusion operations. Figure 5 As shown, two consecutive convolution operations are performed on the target downsampling result. The convolution result is then deconvolved once to obtain a deconvolution result. This deconvolution result is then fused with the convolution result from the corresponding downsampling operation to obtain a set of upsampling results. Performing N consecutive upsampling operations yields the target upsampling result. The number of convolutions included in a set of deconvolution operations in this embodiment can be determined based on actual conditions. The above example is merely illustrative and is not intended to limit the scope. For instance, a set of deconvolution operations could also include three convolutions and two deconvolutions.

[0055] In the above embodiments, depth estimation is performed on the image to be detected captured by the mobile terminal using a depth estimation model, thereby obtaining the distance between the object corresponding to each pixel in the image and the mobile terminal. By capturing an image of the object using the camera on the mobile terminal, the distance between the object and the mobile device can be obtained, thus determining the distance between the obstacle and the user. This method, which obtains the distance between the obstacle and the user simply by capturing an image with the mobile terminal, avoids the resource waste problem of combining data from multiple sensors for obstacle detection in existing technologies, achieving a cost-saving effect.

[0056] Optionally, the step of iteratively performing N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected using the depth estimation model to obtain the target downsampling result includes: performing the i-th set of downsampling operations through the following steps, wherein the initial value of the previous downsampling result is the pixel value of the pixel in the image to be detected, 1≤i≤N, and the target downsampling result is the N-th downsampling result: performing multiple convolution operations on the previous downsampling result using the i-th convolution kernel in an iterative manner to obtain the i-th convolution result; performing the pooling operation in the i-th set of downsampling operations on the i-th convolution result to obtain the i-th downsampling result.

[0057] As an optional implementation, such as Figure 6 The diagram shown is a schematic representation of a downsampling operation structure according to an optional embodiment of the present invention. Figure 1 ,exist Figure 6 Each downsampling operation shown includes two convolution operations and one pooling operation, to Figure 6Taking the illustrated downsampling operation structure as an example, the image to be detected is input into the depth estimation model. Two convolutions and one downsampling operation are performed to obtain the first downsampling result. Two convolution operations and a first pooling operation are then performed on the first downsampling result to obtain the second convolution result. This downsampling operation is repeated N times to obtain the target downsampling result. In each downsampling operation, a convolution operation is performed on the previous downsampling result to obtain the convolution result, and a pooling operation is performed on the convolution result to obtain the current downsampling result. For the convolution operation in the downsampling operation, an a×a convolution kernel can be used. The size of 'a' can be determined according to the actual situation. For example, a 3×3 convolution kernel can be used to iteratively perform convolution operations on the previous downsampling result to obtain the convolution result. In this embodiment, the convolution kernel size in different groups of downsampling operations can be the same or different. For example, the convolution kernel size in the first group of downsampling operations and the convolution kernel size in the second group of downsampling operations can be the same or different, such as a 3×3 convolution kernel in the first group of downsampling operations and a 2×2 convolution kernel in the second group of downsampling operations. The kernel size of convolutions in different iterations within the same set of downsampling operations can be the same or different. For example, in the first set of downsampling operations, the kernel size used for the first and second convolutions can be the same or different. For instance, the kernel size for the first convolution in the first set of downsampling operations could be 3×3, and the kernel size for the second convolution could be 2×2. In this embodiment, by performing convolution and pooling downsampling operations on the image to be detected, N levels of features representing different semantic levels can be obtained. Combined with the upsampling operation in the depth estimation model, a depth map matching the image to be detected can be obtained. The depth map can be used to predict the distance between the user and obstacles, thereby alerting blind users to the location of obstacles.

[0058] Optionally, the previous downsampling result includes A i-1 ×B i-1 ×C i-1 Feature data, where A i-1 B represents the height of the image obtained after the last downsampling operation. i-1 C represents the width of the image obtained after performing the previous downsampling operation. i-1 This represents the number of channels in the image obtained by performing the previous downsampling operation; the step of performing multiple convolution operations on the previous downsampling result using the i-th convolution kernel in an iterative manner to obtain the i-th convolution result includes: using the i-th convolution kernel to perform multiple convolution operations on the image obtained by the previous downsampling operation. i-1 ×B i-1 ×C i-1 The feature data is subjected to Q convolution operations in an iterative manner to obtain the i-th convolution result, wherein the i-th convolution result includes (A i-1 -Q×M i )×(B i-1 -Q×Mi )×C i-1 Feature data, M i Corresponding to the size of the i-th convolutional kernel; the step of performing the pooling operation in the i-th group of downsampling operations on the i-th convolutional result to obtain the i-th downsampling result includes: for the (A i-1 -Q×M i )×(B i-1 -Q×M i )×C i-1 The feature data is subjected to pooling operation in the i-th group of downsampling operations to obtain the i-th downsampling result, wherein the i-th downsampling result includes A i ×B i ×C i Feature data, A i =(A i-1 -Q×M i ) / P i B i = (B i-1 -Q×M i ) / P i C i =C i-1 ×P i P i It corresponds to the size of the pooling window used in the pooling operation of the i-th group of downsampling operations.

[0059] As an optional implementation, let the previous downsampling result be A. i-1 ×B i-1 ×C i-1 Taking the feature data as an example, where A i-1 B represents the height of the image in the (i-1)th downsampling result. i-1 C represents the width of the image in the (i-1)th downsampling result. i-1 This represents the number of channels in the (i-1)th downsampling result. The convolution operation in the i-th downsampling operation changes the height and width of the image in the (i-1)th downsampling result. The magnitude of the change in image width and height is related to the size of the convolution kernel, resulting in the i-th convolution result. Similarly, the pooling operation in the i-th downsampling operation changes the image width, height, and number of channels in the i-th convolution result. The magnitude of the change in image width, height, and number of channels is related to the size of the pooling window in the pooling operation.

[0060] Figure 7 This is a schematic diagram of an optional downsampling operation structure according to an embodiment of the present invention. Figure 2 ,by Figure 7Taking the illustrated downsampling operation structure as an example, the (i-1)th downsampling result includes 140×140×256 feature data, where 140 represents the image width and height in the (i-1)th downsampling result, and 256 represents the number of channels. In the i-th downsampling operation, the first convolution operation is performed on the 140×140×256 feature data to obtain 138×138×256 feature data, the second convolution operation is performed on the 138×138×256 feature data to obtain 136×136×256 feature data, and the pooling operation is performed on the 136×136×256 feature data to obtain the i-th downsampling result, which is 68×68×512 feature data. In this embodiment, the width, height, and number of channels in the above feature data are only for illustrative purposes. The specific size is related to factors such as the size of the image to be detected, the number of convolutions, the size of the convolution kernel, the number of pooling operations, and the pooling window, and is not limited here. In this embodiment, by performing convolution and pooling operations on the feature data in the image to be detected, a depth map matching the image to be detected can be obtained. The distance between the object corresponding to each pixel in the image to be detected and the mobile terminal can be estimated. Obstacle detection can be achieved using only images captured by the mobile terminal, thereby reducing costs. Furthermore, blind people can navigate using only terminals with shooting functions such as mobile phones and tablets, providing convenience for blind users.

[0061] Optionally, the step of iteratively performing N sets of upsampling operations on the target downsampling result using the depth estimation model to obtain the target upsampling result includes: performing the j-th set of upsampling operations through the following steps, wherein the initial value of the previous upsampling result is the target downsampling result, 1≤j≤N, and the target upsampling result is the N-th upsampling result: performing multiple convolution operations on the previous upsampling result using the j-th convolution kernel in an iterative manner to obtain the j-th convolution result; performing the deconvolution operation in the j-th set of upsampling operations on the j-th convolution result to obtain the j-th deconvolution result; and performing feature fusion on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th set of downsampling operations to obtain the j-th upsampling result.

[0062] As an optional implementation, Figure 5 Taking the upsampling shown as an example, in Figure 5Each upsampling operation shown includes two convolution operations, one deconvolution operation, and one feature fusion. In each upsampling operation, a convolution operation is performed on the previous upsampling result to obtain the convolutional result. A deconvolution operation is then performed on the convolutional result to obtain the deconvolutional result. The deconvolutional result is then fused with the convolutional result obtained in the corresponding downsampling operation to obtain the current upsampling result. For the convolution operation in the upsampling operation, a b×b convolutional kernel can be used. The size of b can be determined according to the actual situation. For example, a 4×4 convolutional kernel can be used to iteratively perform convolution operations on the previous upsampling result to obtain the convolutional result. For the deconvolution operation in the upsampling operation, a c×c convolutional kernel can be used. The size of c can be determined according to the actual situation. For example, a 5×5 convolutional kernel can be used to deconvolve the convolutional result to obtain the deconvolutional result. In this embodiment, for convolution and deconvolution, the kernel size in different groups of upsampling operations can be the same or different. For example, the kernel size in the first group of upsampling operations and the kernel size in the second group of upsampling operations can be the same or different, such as the kernel size being 3×3 in the first group of upsampling operations and 2×2 in the second group of upsampling operations. The kernel size in different iterations within the same group of upsampling operations can also be the same or different. For example, in the first group of upsampling operations, the kernel size used for the first and second convolutions can be the same or different. In this embodiment, performing convolution, deconvolution, and feature fusion on the target downsampling results yields an N-level inverse depth map. The depth map can be used to estimate the distance between the user and obstacles, thus alerting blind users to the location of obstacles.

[0063] Optionally, the previous upsampling result includes A j-1 ×B j-1 ×C j-1 Feature data, where A j-1 B represents the height of the image obtained after the last upsampling operation. j-1 C represents the width of the image obtained after performing the previous upsampling operation. j-1 This represents the number of channels in the image obtained by performing the previous upsampling operation; the step of performing multiple convolution operations on the previous upsampling result using the j-th convolution kernel in an iterative manner to obtain the j-th convolution result includes: using the j-th convolution kernel to perform multiple convolution operations on the image obtained by the previous upsampling operation. j-1 ×B j-1 ×C j-1 The feature data is subjected to H convolution operations in an iterative manner to obtain the j-th convolution result, wherein the j-th convolution result includes (A j-1 -H×M j )×(B j-1 -H×M j)×(C j-1 -H×T j Feature data of M j and T j Corresponding to the size of the j-th convolution kernel; the step of performing the deconvolution operation in the j-th group of upsampling operations on the j-th convolution result to obtain the j-th deconvolution result includes: performing the deconvolution operation on the (A) kernel. j-1 -H×M j )×(B j-1 -H×M j )×(C j-1 -H×T j The feature data of ) undergoes a deconvolution operation in the j-th upsampling operation to obtain the j-th deconvolution result, wherein the j-th upsampling result includes D j ×E j ×F j Feature data, D j =(A j-1 -H×M j )×P j E j = (B j-1 -H×M j )×P j F j =C j-1 -H×T j P j It corresponds to the size of the third convolution kernel used in the deconvolution operation of the j-th group of upsampling operations.

[0064] As an optional implementation, let's take the previous upsampling result as A. j-1 ×B j-1 ×C j-1 Taking the feature data as an example, where A j-1 B represents the height of the image in the (j-1)th upsampling result. j-1 C represents the width of the image in the (j-1)th upsampling result. j-1 This represents the number of channels in the (j-1)th upsampling result. Through the convolution operation in the j-th downsampling operation, the height, width, and number of channels of the image in the (j-1)-th upsampling result can be changed. The magnitude of the change is related to the size of the convolution kernel, thus obtaining the j-th convolution result. Through the deconvolution operation in the j-th upsampling operation, the width, height, and number of channels of the image in the j-th convolution result can be changed. The magnitude of the change is related to the size of the deconvolution kernel.

[0065] Figure 8 This is a schematic diagram of an optional upsampling operation structure according to an embodiment of the present invention. Figure 8Taking the upsampling shown as an example, the (j-1)th upsampling result in the figure includes 104×104×512 feature data. In the j-th group of upsampling operations, the first convolution operation is performed on the 104×104×512 feature data to obtain the first convolution result including 102×102×256 feature data. The first convolution result is then convolved to obtain the second convolution result including 100×100×128 feature data. The second convolution result is then deconvolved to obtain the j-th deconvolution result in the j-th group of upsampling operations. The j-th deconvolution result includes 200×200×128 feature data.

[0066] Optionally, feature fusion is performed on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th downsampling operation to obtain the j-th upsampling result, including: selecting D from the convolution results obtained in the (N-j+1)-th downsampling operation. j ×E j ×F j The feature data is used to obtain the j-th group of data to be fused; the j-th group of data to be fused and the j-th deconvolution result are fused to obtain the j-th upsampling result, wherein the j-th upsampling result includes A j ×B j ×C j Feature data.

[0067] As an optional implementation, Figure 8 Taking the upsampling structure diagram as an example, the j-th deconvolution result obtained in the j-th upsampling operation includes 200×200×128 feature data. In the convolution result obtained in the (N-j+1)-th downsampling operation corresponding to the j-th upsampling operation, 200×200×128 feature data is selected and fused with the 200×200×128 feature data included in the deconvolution result and the 200×200×128 feature data included in the j-th deconvolution result to obtain 200×200×256 feature data.

[0068] Optionally, determining the depth estimation result of the image to be detected based on the target upsampling result includes: inputting the target upsampling result into a fully connected layer in the depth estimation model, and obtaining a distance estimate corresponding to the pixel value of each pixel in the target upsampling result through the fully connected layer. The depth estimation model is a model trained using a set of sample data on an original neural network model. The set of sample data includes sample images and a known distance between the object represented by each pixel in the sample images and the sample terminal. The sample images are images captured by the sample terminal. The distance estimate of the sample images output by the fully connected layer in the trained depth estimation model satisfies a preset convergence condition with the known distance. The height of the image in the target upsampling result is the same as the height of the image to be detected, and the width of the image in the target upsampling result is the same as the width of the image to be detected.

[0069] As an optional implementation, the target downsampling result is iteratively upsampled N times to obtain the target upsampling result. The target upsampling result can be a depth map with the same size as the image to be detected, such as... Figure 9 The diagram illustrates a depth profile according to an optional embodiment of the present invention. The pixel value of each pixel in the depth map corresponds to the distance between each pixel in the image to be detected and the mobile terminal. Specifically, the distance corresponding to each pixel in the depth map can be estimated using a fully connected layer of a depth estimation model. This depth estimation model is obtained through repeated training using training data. The training data includes sample images and the known distance between each pixel in the sample images and the location of the mobile terminal when the sample images were captured. The estimated distance between each pixel in the sample images and the mobile terminal output by the fully connected layer of the trained depth estimation model satisfies a convergence condition with the known distance. The convergence condition can be that the output value of the convergence function between the estimated distance and the known distance is within a preset range. The convergence function can be selected according to the actual situation, and the preset range can also be determined according to the actual situation. In this embodiment, the depth of each pixel in the depth map can be estimated using the fully connected layer of the depth estimation model. Since the pixels in the depth map correspond to the pixels in the image to be detected, the distance between the object corresponding to the pixel in the image to be detected and the mobile terminal can be obtained. In this embodiment, the depth estimation model estimates the distance between the object corresponding to each pixel in the image to be detected captured by the mobile terminal and the location of the mobile terminal when the image to be detected was captured. This can avoid the high cost caused by using multiple sensors and achieve the goal of saving costs.

[0070] Optionally, the image to be detected is input into an obstacle detection model to obtain an obstacle recognition result for the image to be detected, including: recognizing the object to be identified in the image to be detected to obtain multiple detection boxes, wherein the detection boxes are used to represent the region of the object to be identified in the image to be detected; determining a target detection box from the multiple detection boxes, wherein the object to be identified within the target detection box is an obstacle of the target type; and outputting the region of the target detection box in the image to be detected, and the target type.

[0071] As an optional implementation, the obstacle detection model is trained on a neural network model using multiple sets of training data. The training data includes obstacles and their types, such as people and vehicles. The obstacle detection model can identify specific types of obstacles, and the types of obstacles it can identify are related to the obstacle types in the training samples. For example, an obstacle detection model trained using people and vehicles can detect people and vehicles in the image to be detected. In this embodiment, the image to be detected is input into the obstacle detection model, which can label objects appearing in the image with detection boxes and identify the target type of obstacle within the labeled detection boxes. Figure 10 This is an optional schematic diagram of an image to be detected according to an embodiment of the present invention. Figure 10 Taking the image to be detected as an example, inputting this image into the obstacle detection model allows the model to label objects in the image using bounding boxes. Figure 10 People and vehicles appearing in the scene are labeled. The obstacle detection model identifies target detection boxes from the labeled multiple detection boxes. The objects to be identified within the target detection boxes are obstacles of the target type. Figure 10 In the image to be detected shown, the obstacle detection model can identify the person in the image as an obstacle. The obstacle detection model can output the region of the obstacle in the image to be detected, as well as the type of obstacle. In this embodiment, the obstacle detection model can accurately identify the region of the obstacle in the image to be detected, as well as the type of obstacle. Combined with the distance estimate of each pixel in the image to be detected output by the depth estimation model, the distance between the obstacle in the image to be detected and the mobile terminal can be obtained.

[0072] Optionally, determining the distance between the obstacle of the target type and the mobile terminal based on the obstacle recognition result and the depth estimation result includes: matching the obstacle recognition result with the depth estimation result, determining a set of depth estimation values ​​from the depth estimation result, wherein the set of depth estimation values ​​is used to represent the distance between the object corresponding to each pixel of the obstacle of the target type in the image to be detected and the mobile terminal; and determining the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimation values.

[0073] As an optional implementation, such as Figure 11 The diagram illustrates the overall model structure according to an optional embodiment of the present invention. It includes a depth estimation model and an obstacle detection model. Inputting the image to be detected into the depth estimation model yields a depth map, where the pixel value of each pixel in the depth map represents the distance between the corresponding pixel in the image to be detected and the mobile terminal. Inputting the image to be detected into the obstacle detection model allows for the labeling of obstacles in the image using detection bounding boxes. By comparing the depth map obtained from the depth estimation model with the region of the obstacle in the image to be detected identified by the obstacle detection model, the distance between the obstacle and the mobile terminal can be obtained. In this embodiment, by matching the obstacle identification result with the depth estimation result, the distance between the obstacle and the mobile terminal can be obtained, achieving obstacle detection using only images captured by the mobile terminal.

[0074] Optionally, determining the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimates includes: determining the smallest depth estimate in the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal; or determining the mean of the depth estimates in the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal.

[0075] As an optional implementation, matching the depth map with obstacle detection results can determine the region of the obstacle in the depth map. The pixel values ​​of the pixels in this region correspond to a set of depth estimates. The minimum depth estimate can be used as the distance between the obstacle and the mobile terminal, or the average of the set of depth estimates can be used as the distance between the obstacle and the mobile terminal, or a depth estimate can be randomly selected from the set of depth estimates as the distance between the obstacle and the mobile terminal. In this embodiment, the distance between the obstacle and the mobile terminal can be predicted using a depth estimation model and an obstacle detection model. The distance estimate is used as the distance between the user using the mobile terminal and the obstacle, thereby achieving the purpose of navigation for the blind. Blind users can achieve navigation using only a mobile phone, achieving a convenient and easy-to-operate effect.

[0076] Optionally, the method further includes: displaying on the mobile terminal the detection area of ​​the obstacle of the target type and the distance between the obstacle of the target type and the mobile terminal; and / or broadcasting on the mobile terminal the orientation of the obstacle of the target type relative to the mobile terminal and the distance between the obstacle of the target type and the mobile terminal.

[0077] As an optional embodiment, the location of the obstacle and the distance between the obstacle and the mobile terminal can be displayed on the mobile terminal, or the location of the obstacle and the distance between the obstacle and the mobile terminal can be announced via voice. Figure 12 The image shown is an optional interface illustration. Figure 3 This interface can simultaneously display obstacle type, direction, and distance, and can also automatically play the obstacle's location, type, and distance via voice. In this embodiment, by visually displaying the obstacle recognition results and playing the obstacle's direction and distance via voice, multiple ways to display obstacle recognition results can be provided to users, enhancing the user experience.

[0078] Optionally, in this embodiment, the terminal device can be a terminal device configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client may be a video client, instant messaging client, browser client, educational client, etc. The network may include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made in this embodiment.

[0079] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0080] According to another aspect of the present invention, an obstacle detection apparatus for implementing the above-described obstacle detection method is also provided. For example... Figure 13 As shown, the device includes: an acquisition module 1302, used to acquire an image to be detected captured by a mobile terminal; a first input module 1304, used to input the image to be detected into an obstacle detection model to obtain an obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify obstacles of a target type in the image to be detected, and the obstacle recognition result is used to represent the region of the identified obstacle of the target type in the image to be detected; a second input module 1306, used to input the image to be detected into a depth estimation model to obtain a depth estimation result of the image to be detected, wherein the depth estimation model is used to perform depth estimation for each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal; and a determination module 1308, used to determine the distance between the obstacle of the target type and the mobile terminal based on the obstacle recognition result and the depth estimation result.

[0081] Optionally, the above-described apparatus is further configured to input the image to be detected into the depth estimation model to obtain the depth estimation result of the image to be detected in the following manner: The depth estimation model iteratively performs N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected to obtain a target downsampling result, wherein N≥2 and N is a natural number, each set of downsampling operations includes a convolution operation and a pooling operation, and the convolution operation in each set of downsampling operations yields a corresponding convolution result; The depth estimation model iteratively performs N sets of upsampling operations on the target downsampling result to obtain a target upsampling result, wherein each set of upsampling operations includes a convolution operation, a deconvolution operation, and a feature fusion operation, and the feature fusion operation in each set of upsampling operations is used to fuse the deconvolution result obtained by the deconvolution operation and the convolution result obtained in the corresponding set of downsampling operations; The depth estimation result of the image to be detected is determined based on the target upsampling result.

[0082] Optionally, the above-described apparatus is further configured to perform N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected in an iterative manner using the depth estimation model to obtain the target downsampling result in the following manner: The i-th set of downsampling operations is performed through the following steps, wherein the initial value of the previous downsampling result is the pixel value of the pixel in the image to be detected, 1≤i≤N, and the target downsampling result is the N-th downsampling result: Multiple convolution operations are performed iteratively on the previous downsampling result using the i-th convolution kernel to obtain the i-th convolution result; Pooling operations are performed on the i-th convolution result in the i-th set of downsampling operations to obtain the i-th downsampling result.

[0083] Optionally, the previous downsampling result includes A. i-1 ×B i-1 ×C i-1 Feature data, where A i-1 B represents the height of the image obtained after the last downsampling operation. i-1 C represents the width of the image obtained after performing the previous downsampling operation. i-1 This indicates the number of channels in the image obtained by performing the previous downsampling operation; the above apparatus is also used to apply the i-th convolution kernel to the A... i-1 ×B i-1 ×C i-1 The feature data is subjected to Q convolution operations in an iterative manner to obtain the i-th convolution result, wherein the i-th convolution result includes (A i-1 -Q×M i )×(B i-1 -Q×M i )×C i-1 Feature data, M i Corresponding to the size of the i-th convolutional kernel; for the (A i-1 -Q×M i )×(B i-1 -Q×M i )×C i-1 The feature data is subjected to pooling operation in the i-th group of downsampling operations to obtain the i-th downsampling result, wherein the i-th downsampling result includes A i ×B i ×C i Feature data, A i =(A i-1 -Q×M i ) / P i B i = (B i-1 -Q×M i ) / P i C i =C i-1 ×P i P i It corresponds to the size of the pooling window used in the pooling operation of the i-th group of downsampling operations.

[0084] Optionally, the above-described apparatus is further configured to perform N sets of upsampling operations iteratively on the target downsampling result using the depth estimation model to obtain the target upsampling result in the following manner: The j-th set of upsampling operations is performed through the following steps, wherein the initial value of the previous upsampling result is the target downsampling result, 1≤j≤N, and the target upsampling result is the N-th upsampling result: Multiple convolution operations are performed iteratively on the previous upsampling result using the j-th convolution kernel to obtain the j-th convolution result; the deconvolution operation in the j-th set of upsampling operations is performed on the j-th convolution result to obtain the j-th deconvolution result; feature fusion is performed on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th set of downsampling operations to obtain the j-th upsampling result.

[0085] Optionally, the previous upsampling result includes A j-1 ×B j-1 ×C j-1 Feature data, where A j-1 B represents the height of the image obtained after the last upsampling operation. j-1 C represents the width of the image obtained after performing the previous upsampling operation. j-1 This indicates the number of channels in the image obtained by performing the previous upsampling operation; the above apparatus is also used to apply the j-th convolution kernel to the A... j-1 ×B j-1 ×C j-1 The feature data is subjected to H convolution operations in an iterative manner to obtain the j-th convolution result, wherein the j-th convolution result includes (A j-1 -H×M j )×(B j-1 -H×M j )×(C j-1 -H×T j Feature data of M j and T j Corresponding to the size of the j-th convolutional kernel; for the (A j-1 -H×M j )×(B j-1 -H×M j )×(C j-1 -H×T j The feature data of ) undergoes a deconvolution operation in the j-th upsampling operation to obtain the j-th deconvolution result, wherein the j-th upsampling result includes D j ×E j ×F j Feature data, D j =(A j-1 -H×M j )×Pj E j = (B j-1 -H×M j )×P j F j =C j-1 -H×T j P j It corresponds to the size of the third convolution kernel used in the deconvolution operation of the j-th group of upsampling operations.

[0086] Optionally, the above-described apparatus is further used to select D from the convolution results obtained in the (N-j+1)th group of downsampling operations. j ×E j ×F j The feature data is used to obtain the j-th group of data to be fused; the j-th group of data to be fused and the j-th deconvolution result are fused to obtain the j-th upsampling result, wherein the j-th upsampling result includes A j ×B j ×C j Feature data.

[0087] Optionally, the above-described apparatus is further configured to input the target upsampling result into the fully connected layer of the depth estimation model, and obtain a distance estimate corresponding to the pixel value of each pixel in the target upsampling result through the fully connected layer. The depth estimation model is a model trained using a set of sample data on an original neural network model. The set of sample data includes sample images and a known distance between the object represented by each pixel in the sample images and the sample terminal. The sample images are images captured by the sample terminal. The distance estimate of the sample images output by the fully connected layer in the trained depth estimation model satisfies a preset convergence condition with the known distance. The height of the image in the target upsampling result is the same as the height of the image to be detected, and the width of the image in the target upsampling result is the same as the width of the image to be detected.

[0088] Optionally, the above-described apparatus is further configured to identify an object to be identified in the image to be detected, thereby obtaining multiple detection boxes, wherein the detection boxes represent the region of the object to be identified in the image to be detected; determine a target detection box from the multiple detection boxes, wherein the object to be identified within the target detection box is an obstacle of the target type; and output the region of the target detection box in the image to be detected, as well as the target type.

[0089] Optionally, the above-described apparatus is further configured to match the obstacle recognition result with the depth estimation result, determine a set of depth estimation values ​​from the depth estimation result, wherein the set of depth estimation values ​​is used to represent the distance between the object corresponding to each pixel of the obstacle of the target type in the image to be detected and the mobile terminal; and determine the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimation values.

[0090] Optionally, the above-described device is further configured to determine the minimum depth estimate among the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal; and to determine the mean of the depth estimates among the set of depth estimates as the distance between the obstacle of the target type and the mobile terminal.

[0091] Optionally, the above-described device is further configured to display on the mobile terminal the detection area of ​​the obstacle of the target type and the distance between the obstacle of the target type and the mobile terminal; and to broadcast on the mobile terminal the orientation of the obstacle of the target type relative to the mobile terminal and the distance between the obstacle of the target type and the mobile terminal.

[0092] According to another aspect of the present invention, an electronic device for implementing the above-described obstacle detection method is also provided, the electronic device being... Figure 1 The terminal device or server shown. This embodiment uses electronic device 14 as an example for illustration. Figure 14 As shown, the electronic device includes a memory 1402 and a processor 1404. The memory 1402 stores a computer program, and the processor 1404 is configured to execute the steps of any of the above method embodiments via the computer program.

[0093] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0094] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0095] S1, acquire the image to be detected captured by the mobile terminal;

[0096] S2, the image to be detected is input into the obstacle detection model to obtain the obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify the target type of obstacle in the image to be detected, and the obstacle recognition result is used to represent the area of ​​the identified target type obstacle in the image to be detected;

[0097] S3, input the image to be detected into the depth estimation model to obtain the depth estimation result of the image to be detected, wherein the depth estimation model is used to perform depth estimation for each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal;

[0098] S4. Based on the obstacle identification result and the depth estimation result, determine the distance between the target type obstacle and the mobile terminal.

[0099] Alternatively, as those skilled in the art will understand, Figure 14 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 14 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 14 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 14 The different configurations shown.

[0100] The memory 1402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the obstacle detection method and device in this embodiment of the invention. The processor 1404 executes various functional applications and data processing by running the software programs and modules stored in the memory 1402, thereby realizing the obstacle detection method described above. The memory 1402 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1402 may further include memory remotely located relative to the processor 1404, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1402 may be used, but is not limited to, to store information such as sample characteristics of items and target virtual resource accounts. As an example, such as Figure 14 As shown, the memory 1402 may include, but is not limited to, the acquisition module 1302, the first input module 1304, the second input module 1306, and the determination module 1308 from the obstacle detection device. Furthermore, it may include, but is not limited to, other module units from the obstacle detection device, which will not be elaborated upon in this example.

[0101] Optionally, the transmission device 1406 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1406 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1406 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0102] In addition, the aforementioned electronic device also includes: a display 1408 for displaying the image to be detected, and the distance between the target type obstacle and the mobile terminal; and a connection bus 1410 for connecting the various module components in the aforementioned electronic device.

[0103] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0104] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above. The computer program is configured to perform the steps in any of the method embodiments described above at runtime.

[0105] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0106] S1, acquire the image to be detected captured by the mobile terminal;

[0107] S2, the image to be detected is input into the obstacle detection model to obtain the obstacle recognition result of the image to be detected, wherein the obstacle detection model is used to identify the target type of obstacle in the image to be detected, and the obstacle recognition result is used to represent the area of ​​the identified target type obstacle in the image to be detected;

[0108] S3, input the image to be detected into the depth estimation model to obtain the depth estimation result of the image to be detected, wherein the depth estimation model is used to perform depth estimation for each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal;

[0109] S4. Based on the obstacle identification result and the depth estimation result, determine the distance between the target type obstacle and the mobile terminal.

[0110] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0111] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0113] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting obstacles, characterized in that, include: Acquire the image to be detected captured by the mobile terminal; The image to be detected is input into the obstacle detection model to obtain detection boxes that match each of the multiple obstacles in the image to be detected. Target detection boxes matching target obstacles of the target type are determined from a plurality of detection boxes, wherein the target detection boxes are used to indicate the region of the target obstacle of the target type in the image to be detected; The image to be detected is input into a depth estimation model to obtain a depth estimation result for each pixel in a depth map that matches the image to be detected. The depth estimation model is used to estimate the depth of each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal. Based on the region of the target detection box in the depth map, a set of depth estimates are determined; Based on the set of depth estimates, the distance between the target type obstacle and the mobile terminal is determined.

2. The method according to claim 1, characterized in that, The image to be detected is input into a depth estimation model to obtain a depth estimation result for the image to be detected, including: The depth estimation model iteratively performs N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected to obtain the target downsampling result, where N≥2 and N is a natural number. Each set of downsampling operations includes convolution and pooling operations. The convolution operation in each set of downsampling operations is executed to obtain the corresponding convolution result. The target downsampling result is iteratively upsampled N times by the depth estimation model to obtain the target upsampling result. Each upsampling operation includes a convolution operation, a deconvolution operation and a feature fusion operation. The feature fusion operation in each upsampling operation is used to fuse the features of the deconvolution result obtained by the deconvolution operation and the convolution result obtained in the corresponding set of downsampling operations. Based on the target upsampling results, the depth estimation result of the image to be detected is determined.

3. The method according to claim 2, characterized in that, The step of iteratively performing N sets of downsampling operations on the pixel values ​​of the pixels in the image to be detected using the depth estimation model to obtain the target downsampling result includes: The i-th downsampling operation is performed through the following steps, wherein the initial value of the previous downsampling result is the pixel value of the pixel in the image to be detected, 1≤i≤N, and the target downsampling result is the N-th downsampling result: The i-th convolution kernel is used to perform multiple convolution operations on the previous downsampling result in an iterative manner to obtain the i-th convolution result; The pooling operation in the i-th group of downsampling operations is performed on the i-th convolution result to obtain the i-th downsampling result.

4. The method according to claim 3, characterized in that, The previous downsampling result includes A i-1 ×B i-1 ×C i-1 Feature data, where A i-1 B represents the height of the image obtained after the last downsampling operation. i-1 C represents the width of the image obtained after performing the previous downsampling operation. i-1 This indicates the number of channels in the image obtained after performing the previous downsampling operation; The step of performing multiple convolution operations on the previous downsampling result using the i-th convolution kernel in an iterative manner to obtain the i-th convolution result includes: using the i-th convolution kernel to perform multiple convolution operations on the A... i-1 ×B i-1 ×C i-1 The feature data is subjected to Q convolution operations in an iterative manner to obtain the i-th convolution result, wherein the i-th convolution result includes (A i-1 -Q×M i )×(B i-1 - Q×M i )×C i-1 Feature data, M i Corresponding to the size of the i-th convolutional kernel; The step of performing the pooling operation in the i-th group of downsampling operations on the i-th convolution result to obtain the i-th downsampling result includes: performing pooling operations on the (A)-th group of downsampling operations on the i-th convolution result to obtain the i-th downsampling result. i-1 - Q×M i )×(B i-1 - Q×M i )×C i-1 The feature data is subjected to pooling operation in the i-th group of downsampling operations to obtain the i-th downsampling result, wherein the i-th downsampling result includes A i ×B i ×C i Feature data, A i = (A i-1 - Q×M i ) / P i B i = (B i-1 - Q×M i ) / P i C i = C i-1 ×P i P i It corresponds to the size of the pooling window used in the pooling operation of the i-th group of downsampling operations.

5. The method according to claim 2, characterized in that, The step of iteratively performing N sets of upsampling operations on the target downsampling results using the depth estimation model to obtain the target upsampling results includes: The j-th upsampling operation is performed through the following steps, wherein the initial value of the previous upsampling result is the target downsampling result, 1≤j≤N, and the target upsampling result is the N-th upsampling result: The j-th convolution kernel is used to perform multiple convolution operations on the previous upsampling result in an iterative manner to obtain the j-th convolution result; Perform the deconvolution operation in the j-th upsampling operation on the j-th convolution result to obtain the j-th deconvolution result; The j-th deconvolution result and the convolution result obtained in the N-j+1th downsampling operation are fused to obtain the j-th upsampling result.

6. The method according to claim 5, characterized in that, The previous upsampling result includes A j-1 ×B j-1 ×C j-1 Feature data, where A j-1 B represents the height of the image obtained after the last upsampling operation. j-1 C represents the width of the image obtained after performing the previous upsampling operation. j-1 This indicates the number of channels in the image obtained after performing the previous upsampling operation; The step of performing multiple convolution operations on the previous upsampling result using the j-th convolution kernel in an iterative manner to obtain the j-th convolution result includes: using the j-th convolution kernel to perform multiple convolution operations on the A... j-1 ×B j-1 ×C j-1 The feature data is subjected to H convolution operations in an iterative manner to obtain the j-th convolution result, wherein the j-th convolution result includes (A j-1 -H×M j )×(B j-1 - H×M j )×(C j-1 - H×T j Feature data of M j and T j Corresponding to the size of the j-th convolutional kernel; The step of performing the deconvolution operation in the j-th upsampling operation on the j-th convolution result to obtain the j-th deconvolution result includes: performing the deconvolution operation on the (A)-th convolution result. j-1 -H×M j )×(B j-1 - H×M j )×(C j-1 - H×T j The feature data of ) undergoes a deconvolution operation in the j-th upsampling operation to obtain the j-th deconvolution result, wherein the j-th upsampling result includes D j ×E j ×F j Feature data, D j = (A j-1 -H×M j )×P j E j = (B j-1 - H×M j )×P j F j =C j-1 - H×T j P j It corresponds to the size of the third convolution kernel used in the deconvolution operation of the j-th group of upsampling operations.

7. The method according to claim 6, characterized in that, Feature fusion is performed on the j-th deconvolution result and the convolution results obtained in the (N-j+1)-th downsampling operation to obtain the j-th upsampling result, including: Select D from the convolution results obtained in the N-j+1th group downsampling operation j ×E j ×F j From the feature data, we obtain the j-th group of data to be fused; The j-th group of data to be fused and the j-th deconvolution result are fused to obtain the j-th upsampling result, wherein the j-th upsampling result includes A. j ×B j ×C j Feature data.

8. The method according to claim 2, characterized in that, The step of determining the depth estimation result of the image to be detected based on the target upsampling result includes: The target upsampling result is input into the fully connected layer of the depth estimation model. The fully connected layer obtains the distance estimate corresponding to the pixel value of each pixel in the target upsampling result. The depth estimation model is a model trained on the original neural network model using a set of sample data. The set of sample data includes sample images and the known distance between the object represented by each pixel in the sample images and the sample terminal. The sample images are images taken by the sample terminal. The distance estimate of the sample images output by the fully connected layer in the trained depth estimation model satisfies the preset convergence condition with the known distance. The height of the image in the target upsampling result is the same as the height of the image to be detected, and the width of the image in the target upsampling result is the same as the width of the image to be detected.

9. The method according to claim 1, characterized in that, The image to be detected is input into the obstacle detection model to obtain the obstacle recognition result of the image to be detected, including: The object to be identified in the image to be detected is identified to obtain multiple detection boxes, wherein the detection boxes are used to represent the region of the object to be identified in the image to be detected; A target detection box is determined from the plurality of detection boxes, wherein the object to be identified within the target detection box is an obstacle of the target type; Output the region of the target detection box in the image to be detected, and the target type.

10. The method according to claim 1, characterized in that, Based on the obstacle recognition results and the depth estimation results, the distance between the target type obstacle and the mobile terminal is determined, including: The obstacle recognition result is matched with the depth estimation result, and a set of depth estimation values ​​is determined from the depth estimation result. The set of depth estimation values ​​is used to represent the distance between the object corresponding to each pixel of the target type obstacle in the image to be detected and the mobile terminal. The distance between the target type obstacle and the mobile terminal is determined based on the set of depth estimates.

11. The method according to claim 10, characterized in that, Determining the distance between the target type obstacle and the mobile terminal based on the set of depth estimates includes: The smallest depth estimate among the set of depth estimates is determined as the distance between the obstacle of the target type and the mobile terminal; or The mean of the depth estimates in a set of depth estimates is determined as the distance between the target type obstacle and the mobile terminal.

12. The method according to claim 10, characterized in that, The method further includes: The mobile terminal displays the detection area of ​​the obstacle of the target type, and the distance between the obstacle of the target type and the mobile terminal; and / or The location of the obstacle of the target type relative to the mobile terminal, as well as the distance between the obstacle of the target type and the mobile terminal, are broadcast on the mobile terminal.

13. An obstacle detection device, characterized in that, include: The acquisition module is used to acquire the image to be detected captured by the mobile terminal; The first input module is used to input the image to be detected into the obstacle detection model to obtain detection boxes that match each of the multiple obstacles in the image to be detected; and to determine a target detection box that matches a target obstacle of the target type from the multiple detection boxes, wherein the target detection box is used to indicate the region of the target obstacle of the target type in the image to be detected; The second input module is used to input the image to be detected into the depth estimation model to obtain the depth estimation result of each pixel in the depth map that matches the image to be detected. The depth estimation model is used to perform depth estimation on each pixel in the image to be detected, and the depth estimation result is used to represent the distance between the object represented by each pixel and the mobile terminal. The determination module is used to determine a set of depth estimates based on the region of the target detection box in the depth map; and to determine the distance between the obstacle of the target type and the mobile terminal based on the set of depth estimates.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 12.

15. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 12 through the computer program.

Images