Image detection method, device, electronic device and storage medium

By integrating the global features, local features and depth map features of the image in the three-dimensional vehicle detection, predicting the depth of the obstacle from the camera, solving the problem of insufficient detection accuracy and robustness of the three-dimensional vehicle in the prior art, achieving a higher Detection accuracy and robustness.

CN111784659BInactive Publication Date: 2024-11-12BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010605474.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-29
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing three-dimensional vehicle detection methods cannot meet the requirements of accuracy and robustness at the same time, especially in road scenes, the obstacle depth estimation accuracy of binocular cameras is high and the calculation is complex. The point cloud generated by the radar is sparse and the distance detection accuracy is Low, the 2D image is affected by perspective projection, resulting in the 3D detection frame being inaccurate enough.

Method used

By acquiring the global, local and depth map features of the image to be tested, the depth of the obstacle is predicted from the camera to improve the accuracy and robustness of image detection.

Benefits of technology

Combining the depth map features of the environment to be tested, the depth of obstacles in the image is predicted from the camera, which improves the accuracy and robustness of image detection and meets the accuracy and robustness requirements of three-dimensional vehicle detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111784659B_ABST
    Figure CN111784659B_ABST
Patent Text Reader

Abstract

The present application discloses a method, device, electronic device and storage medium for image detection, which relate to the fields of autonomous driving, computer vision and deep learning. The specific implementation scheme is: obtaining the image to be tested captured by a camera in the environment to be tested; fusing the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested to obtain the fusion features of the image to be tested; and predicting the depth of obstacles in the image to be tested from the camera based on the fusion features of the image to be tested. Compared with the prior art, the present application predicts the depth of obstacles in the image from the camera in combination with the depth map features of the environment to be tested, thereby improving the accuracy and robustness of image detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of computer vision, autonomous driving, and deep learning in computer technology, and in particular, to a method, device, electronic device, and storage medium for image detection. Background Art

[0002] Autonomous driving technology refers to the technology that uses sensors to sense the environment around the vehicle and controls the vehicle's steering and speed based on the road, vehicle position, and obstacles it senses, so that the vehicle can travel safely and reliably on the road. Three-dimensional vehicle detection is used to detect obstacles around the vehicle and is crucial to autonomous driving technology.

[0003] At present, 3D vehicle detection in road scenes is mainly based on images from onboard binocular cameras or radar data. For 3D vehicle detection in fixed monitoring scenes, obstacle detection can be achieved by directly predicting the projection of the eight vertices of the 3D detection box on the image or the length, width, height and orientation angle information through the network; or, by using monocular depth estimation to convert the image combined with depth information into a pseudo point cloud, and then using the 3D point cloud detection method to detect obstacles.

[0004] However, the binocular camera-based approach has high requirements for the accuracy of obstacle depth estimation and high computational complexity, and cannot meet the requirements of real-time and robustness. The radar-based approach does not meet the application scenario requirements of surveillance cameras, and the point cloud generated by the radar is relatively sparse, and the detection accuracy at a distance is low. The two-dimensional image-based approach is often affected by the perspective projection that causes objects to be larger near and smaller far away, resulting in inaccurate estimated 3D detection frames and insufficient detection accuracy. Therefore, existing three-dimensional vehicle detection methods cannot meet the requirements of both accuracy and robustness at the same time. Summary of the invention

[0005] The present application provides an image detection method, device, electronic device and storage medium.

[0006] According to a first aspect of the present application, there is provided an image detection method, comprising:

[0007] Acquire the image to be tested captured by the camera in the environment to be tested;

[0008] Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested;

[0009] The depth of the obstacle in the image to be tested from the camera is predicted according to the fusion features of the image to be tested.

[0010] According to a second aspect of the present application, there is provided an image detection device, comprising:

[0011] An acquisition module is used to acquire the image to be tested collected by the camera in the environment to be tested;

[0012] A fusion module, used to fuse the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested to obtain fusion features of the image to be tested;

[0013] The prediction module is used to predict the depth of the obstacle in the image to be tested from the camera according to the fusion features of the image to be tested.

[0014] According to a third aspect of the present application, an electronic device is provided, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0018] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect above.

[0019] According to a fifth aspect of the present application, there is provided an image detection method, comprising:

[0020] Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested;

[0021] The depth of the obstacle in the image to be tested from the camera is predicted according to the fusion features of the image to be tested.

[0022] The technology of this application solves the technical problem that three-dimensional vehicle detection in the prior art cannot meet the requirements of accuracy and robustness at the same time. Compared with the prior art, this application combines the depth map features of the environment to be tested to predict the depth of obstacles in the image from the camera, thereby improving the accuracy and robustness of image detection.

[0023] According to the sixth aspect of the present application, a computer program product is provided, the program product comprising: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0026] Figure 1 A schematic diagram of a scene of an image detection method provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a flow chart of an image detection method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the principle of image detection provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a flow chart of another image detection method provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a flow chart of another image detection method provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the structure of an image detection device provided in an embodiment of the present application;

[0032] Figure 7 It is a block diagram of an electronic device used to implement the image detection method of an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] At present, 3D vehicle detection in road scenes is mainly based on images from onboard binocular cameras or radar data. For 3D vehicle detection in fixed monitoring scenes, obstacle detection can be achieved by directly predicting the projection of the eight vertices of the 3D detection box on the image or the length, width, height and orientation angle information through the network; or, by using monocular depth estimation to convert the image combined with depth information into a pseudo point cloud, and then using the 3D point cloud detection method to detect obstacles.

[0035] However, the binocular camera-based approach has high requirements for the accuracy of obstacle depth estimation and high computational complexity, and cannot meet the requirements of real-time and robustness. The radar-based approach does not meet the application scenario requirements of surveillance cameras, and the point cloud generated by the radar is relatively sparse, and the detection accuracy at a distance is low. The two-dimensional image-based approach is often affected by the perspective projection that causes objects to be larger near and smaller far away, resulting in inaccurate estimated 3D detection frames and insufficient detection accuracy. Therefore, existing three-dimensional vehicle detection methods cannot meet the requirements of both accuracy and robustness at the same time.

[0036] The present application provides an image detection method, device, electronic device, and storage medium, which are applied to the fields of computer vision, autonomous driving, and deep learning in computer technology to solve the technical problem that three-dimensional vehicle detection cannot simultaneously meet the requirements of accuracy and robustness, thereby achieving the effect of improving the accuracy and robustness of image detection. The inventive concept of the present application is: when performing image detection, depth map features are fused on the basis of existing global features and local features.

[0037] In order to clearly understand the technical solution of this application, the terms involved in this application are explained below:

[0038] Monocular camera: A camera with only one visual sensor.

[0039] Binocular camera: A camera consisting of two visual sensors. The binocular camera can use the principle of triangulation to obtain the depth information of the scene and reconstruct the three-dimensional shape and position of the surrounding scenery.

[0040] Depth map: A depth map, also known as a distance image, is an image that uses the distance from the image collector to each point in the scene as the pixel value.

[0041] Camera intrinsic parameters: parameters related to the camera’s own characteristics, such as the camera’s focal length, pixel size, etc.

[0042] The following describes the usage scenarios of this application.

[0043] Figure 1 A schematic diagram of a scene of an image detection method provided in an embodiment of the present application. Figure 1 The camera 102 on the vehicle 101 collects images of the vehicle's surroundings and sends the images to the server 103. The server 103 detects the images, determines the depth of obstacles in the images from the camera 103, and issues instructions to the vehicle 101 to control the steering and speed of the vehicle 101 for automatic driving.

[0044] The camera 102 may be a monocular camera. The server 102 may be a server or a server in a cloud service platform.

[0045] It should be noted that the application scenarios of the technical solution of this application can be Figure 1 The method is applicable to the autonomous driving scenario in the video, but is not limited to this. It can also be applied to other scenarios that require image detection.

[0046] It can be understood that the above-mentioned image detection method can be implemented by the image detection device provided in the embodiment of the present application. The image detection device can be part or all of a certain device, for example, it can be a server or a processor in the server.

[0047] The following takes a server integrated or installed with relevant execution codes as an example to describe the technical solution of the embodiment of the present application in detail with a specific embodiment. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0048] Figure 2 This is a flow chart of an image detection method provided in an embodiment of the present application. This embodiment relates to the process of how to perform image detection. Figure 2 As shown, the method includes:

[0049] S201, obtaining an image to be tested captured by a camera in a test environment.

[0050] The camera may be a monocular camera, and accordingly, the image to be measured may be a monocular image.

[0051] In this application, the camera can capture the image to be tested in the test environment in real time, and then the server can receive the image to be tested sent by the camera. In the autonomous driving scenario, the camera can be installed at any position of the vehicle. For example, the camera can be installed in front of the vehicle to capture the image to be tested in front of the vehicle; or, the camera can be installed at the rear of the vehicle to capture the image to be tested at the rear of the vehicle.

[0052] This application does not impose any restrictions on the test environment. For example, taking autonomous driving as an example, if the vehicle is in a road scene, the test environment can be a road; if the vehicle is in a fixed monitoring scene, the test environment can be a parking lot under fixed monitoring, etc.

[0053] S202: Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested.

[0054] In this step, after the server obtains the image to be tested captured by the camera in the environment to be tested, the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested may be fused to obtain the fused features of the image to be tested.

[0055] The global feature may be the overall attribute of the image, and may include, for example, color feature, texture feature, shape feature, etc. The global feature has the characteristics of good invariance, simple calculation, intuitive representation, etc.

[0056] A local feature may be a local expression of an image feature. For example, the local feature may include a scale-invariant feature transform (SIFT), a speededup robust features (SURF), a daisy feature, and the like.

[0057] The embodiment of the present application does not limit how to obtain the global features and local features of the image to be tested. In an optional implementation, the image to be tested can be input into the backbone network (Backbone), and the local features and global features output by the Backbone can be obtained. Among them, Backbone is a neural network model for target detection, and exemplary models can be ResNet, DenseNet, etc. The embodiment of the present application does not limit Backbone, and can be specifically set according to actual conditions.

[0058] In an optional implementation, before obtaining the fusion features of the image to be tested, the server extracts the depth map features of the environment to be tested from the ground depth map of the environment to be tested. Similarly, the server can input the ground depth map of the environment to be tested into the backbone network and obtain the depth map features of the environment to be tested output by the backbone network.

[0059] The embodiment of the present application does not limit how to fuse global features, local features and depth map features. For example, if the Backbone used in feature extraction is ResNet, then the element wise-sum method is used for feature fusion. Among them, the element wise-sum method is to combine multiple features into a complex vector.

[0060] For example, if the Backbone used in feature extraction is DenseNet, then the concat method is used to perform feature fusion, where the concat method directly connects multiple features.

[0061] In addition, before extracting the depth map features of the environment to be tested from the ground depth map of the environment to be tested, the server may also determine the depth of at least one point on the ground of the environment to be tested from the camera according to the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system. Subsequently, a ground depth map of the environment to be tested is established according to the depth of at least one point on the ground of the environment to be tested from the camera.

[0062] In this application, features can be extracted quickly and accurately through the backbone network. In addition, since the three-dimensional positions regressed by different cameras are ambiguous, the ambiguity can be eliminated by adding the depth map features of the environment to be tested, and the generalization ability can be improved, thereby meeting the accuracy and robustness requirements of image detection.

[0063] S203: predicting the depth of obstacles in the image to be tested from the camera according to the fusion features of the image to be tested.

[0064] In this step, after the server fuses the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested to obtain the fused features of the image to be tested, the depth of the obstacle in the image to be tested from the camera can be predicted based on the fused features of the image to be tested.

[0065] In some embodiments, the server may input the fusion features of the image to be tested into the neural network model, and obtain the depth of the obstacle from the camera in the image to be tested output by the neural network model.

[0066] Among them, the above neural network model is a convolutional neural network model or a fully connected neural network model.

[0067] It should be noted that the embodiments of the present application do not limit the process of building a neural network model, and the neural network can be built using commonly used convolutional layers or fully connected layers.

[0068] Figure 3 The following is a schematic diagram of the principle of image detection provided by the embodiment of the present application. Figure 3The figure shows a monocular 3D region proposal network for object detection (M3D-RPN). In M3D-RPN, on the one hand, after acquiring the image to be tested, the server inputs the image to be tested into Backbone, and obtains the global features and local features of the image to be tested output by Backbone. On the other hand, the server also inputs the pre-acquired ground depth map into Backbone, and obtains the depth map features of the environment to be tested output by Backbone. Subsequently, the server performs feature fusion on the global features, local features and depth map features, and outputs them to the prediction module for depth prediction to obtain the depth prediction result. In addition, the prediction module can also obtain the original 3D prediction result obtained in the existing M3D-RPN based on the global features and local features, and combine the original 3D prediction result and the depth prediction result into a new 3D prediction result.

[0069] The method of image detection provided in the embodiment of the present application first obtains the image to be tested captured by the camera in the environment to be tested, and then fuses the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain the fusion features of the image to be tested. Finally, based on the fusion features of the image to be tested, the depth of the obstacle in the image to be tested from the camera is predicted. Compared with the prior art, the present application predicts the depth of the obstacle in the image from the camera in combination with the depth map features of the environment to be tested, thereby improving the accuracy and robustness of image detection.

[0070] Based on the above embodiment, how to obtain the depth map features of the environment to be tested is described below. Figure 4 A flowchart of another image detection method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the image detection method includes:

[0071] S301 : Determine a depth of at least one point on the ground of the environment to be measured from a camera according to the coordinates of at least one point on the ground of the environment to be measured in an image coordinate system.

[0072] In some embodiments, the server may obtain the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system, and then determine the depth of at least one point on the ground of the environment to be measured from the camera according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system, the intrinsic parameters of the camera, and the ground equation.

[0073] Specifically, the server may reflect and project the coordinates of at least one point on the ground of the test environment in the image coordinate system according to the intrinsic parameters of the camera and the ground equation, thereby calculating the depth of at least one point on the ground of the test environment from the camera.

[0074] Among them, the camera's intrinsic parameters and ground equations are pre-calibrated, and the coordinates of each point under the image coordinates are known. In the early stage of constructing the ground depth map, it can be assumed that each point is a point on the ground.

[0075] For example, the point on the ground of the environment to be tested is Corner (x, y), the camera internal parameter K is shown in formula (1), and the ground equation is shown in formula (2). Accordingly, the depth of the point on the ground of the environment to be tested from the camera can be calculated by formulas (3)-(5). Formulas (1)-(5) are as follows:

[0076]

[0077] ax+by+cz+d=0........................(2)

[0078]

[0079] point cam =K -1 *Img p ...................................(4)

[0080]

[0081] Among them, a, b, c, d are adjustable parameters, f_x is the number of pixels in the x-axis direction f on the imaging plane with a focal length of f; f_y is the number of pixels in the y-axis direction f on the imaging plane with a focal length of f; c_x is the offset of the origin of the physical imaging plane in the x-axis direction, and c_y is the offset of the origin of the physical imaging plane in the y-axis direction.

[0082] In the present application, through the coordinates of at least one point on the ground of the test environment in the image coordinate system, the camera's internal parameters and the ground equation, the depth of at least one point on the ground from the camera can be quickly and accurately determined, and then the ground depth map of the test environment can be established.

[0083] S302: Establish a ground depth map of the environment to be measured according to the depth of at least one point on the ground of the environment to be measured from a camera.

[0084] In the step, after the server determines the depth of at least one point on the ground of the environment to be tested from the camera based on the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system, it can establish a ground depth map of the environment to be tested based on the depth of at least one point on the ground of the environment to be tested from the camera.

[0085] In some embodiments, the server may use the depth of at least one point on the ground of the environment to be measured from the camera as a pixel value, thereby establishing a ground depth map of the environment to be measured.

[0086] S303: extracting depth map features of the environment to be tested from the ground depth map of the environment to be tested.

[0087] Exemplarily, the server may input a ground depth map of the environment to be tested into the backbone network, and obtain depth map features of the environment to be tested output by the backbone network.

[0088] S304: Acquire an image to be tested captured by a camera in the environment to be tested.

[0089] S305 : Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested.

[0090] S306 : predicting the depth of obstacles in the image to be tested from the camera according to the fusion features of the image to be tested.

[0091] The technical terms, technical effects, technical features, and optional implementation methods of S304-S306 can be found in Figure 2 It is understood that S201-S203 shown in the figure will not be repeated here for the repeated contents.

[0092] Based on the above embodiment, how to predict the depth of an obstacle from a camera in an image is described below. Figure 5 A flowchart of another image detection method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the image detection method includes:

[0093] S401, obtaining an image to be tested captured by a camera in a test environment.

[0094] S402: Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested.

[0095] S403: Input the fusion features of the image to be tested into the neural network model, and obtain the depth of the obstacle from the camera in the image to be tested output by the neural network model.

[0096] Among them, the neural network model is a convolutional neural network model or a fully connected neural network model.

[0097] It should be noted that the embodiments of the present application do not limit the process of building a neural network model, and the neural network can be built using commonly used convolutional layers or fully connected layers.

[0098] In the present application, since the existing neural network model can be used to predict the depth of the obstacle from the camera by fusing features, it is possible to avoid adding additional computing power and improve the timeliness of image detection.

[0099] The method of image detection provided in the embodiment of the present application first obtains the image to be tested captured by the camera in the environment to be tested, and then fuses the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain the fused features of the image to be tested. Finally, the fused features of the image to be tested are input into the neural network model, and the depth of the obstacle from the camera in the image to be tested output by the neural network model is obtained. Compared with the prior art, the present application combines the depth map features of the environment to be tested to predict the depth of the obstacle from the camera in the image, thereby improving the accuracy and robustness of image detection.

[0100] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program information, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0101] Figure 6 The following is a schematic diagram of the structure of an image detection device provided in an embodiment of the present application. The image detection device can be implemented by software, hardware, or a combination of both, such as the above-mentioned server or a chip in the server, for executing the above-mentioned image detection method. Figure 6 As shown, the image detection device 500 includes: an acquisition module 501, a fusion module 502, a prediction module 503, an extraction module 504, a drawing module 505 and a calculation module 506.

[0102] An acquisition module 501 is used to acquire an image to be tested captured by a camera in a test environment;

[0103] A fusion module 502 is used to fuse the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fusion features of the image to be tested;

[0104] The prediction module 503 is used to predict the depth of the obstacle in the image to be tested from the camera according to the fusion features of the image to be tested.

[0105] In an optional implementation manner, the image detection device 500 further includes:

[0106] The extraction module 504 is used to extract the depth map features of the environment to be tested from the ground depth map of the environment to be tested.

[0107] In an optional implementation, the extraction module 504 is specifically used to input the ground depth map of the environment to be tested into the backbone network, and obtain the depth map features of the environment to be tested output by the backbone network.

[0108] In an optional implementation manner, the image detection device 500 further includes:

[0109] The mapping module 505 is used to establish a ground depth map of the environment to be measured according to the depth of at least one point on the ground of the environment to be measured from the camera.

[0110] In an optional implementation manner, the image detection device 500 further includes:

[0111] The calculation module 506 is used to determine the depth of at least one point on the ground of the environment to be measured from the camera according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system.

[0112] In an optional implementation, the calculation module 506 is specifically used to obtain the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system; and determine the depth of at least one point on the ground of the environment to be measured from the camera based on the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system, the intrinsic parameters of the camera and the ground equation.

[0113] In an optional implementation, the calculation module 506 is specifically used to reflect and project the coordinates of at least one point on the ground of the test environment in the image coordinate system according to the intrinsic parameters of the camera and the ground equation, and calculate the depth of at least one point on the ground of the test environment from the camera.

[0114] In an optional implementation, the prediction module 503 is specifically used to input the fusion features of the image to be tested into the neural network model, and obtain the depth of the obstacle from the camera in the image to be tested output by the neural network model.

[0115] In an optional implementation, the neural network model is a convolutional neural network model or a fully connected neural network model.

[0116] In an optional implementation, the image to be measured is a monocular image.

[0117] The image detection device provided in the embodiment of the present application can execute the actions of the image detection method in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0118] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0119] According to an embodiment of the present application, the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0120] like Figure 7 , is a block diagram of an electronic device according to the method of image detection of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0121] like Figure 7 As shown, the electronic device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 601 is taken as an example.

[0122] The memory 602 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the image detection method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the image detection method provided in the present application.

[0123] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the image detection method in the embodiment of the present application (for example, the attached Figure 6The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the image detection method in the above method embodiment is implemented.

[0124] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device for image detection, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the electronic device for image detection via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The electronic device of the image detection method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0126] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for image detection, such as input devices such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0127] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0132] The present application also provides a chip, including a processor and an interface. The interface is used to input and output data or instructions processed by the processor. The processor is used to execute the method provided in the above method embodiment. The chip can be applied to a server.

[0133] The present invention also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program information, and the program information is used for the above method.

[0134] An embodiment of the present application also provides a program, which, when executed by a processor, is used to execute the image detection method provided in the above method embodiment.

[0135] An embodiment of the present application also provides a program product, such as a computer-readable storage medium, in which instructions are stored. When the program product is run on a computer, the computer executes the image detection method provided by the above method embodiment.

[0136] According to the technical solution of the embodiment of the present application, firstly, the image to be tested captured by the camera in the environment to be tested is obtained, and then the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested are fused to obtain the fusion features of the image to be tested. Finally, based on the fusion features of the image to be tested, the depth of the obstacle in the image to be tested from the camera is predicted. Compared with the prior art, the present application predicts the depth of the obstacle in the image from the camera in combination with the depth map features of the environment to be tested, thereby improving the accuracy and robustness of image detection.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0138] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for image detection, include: Acquire an image to be tested captured by a camera in the environment to be tested, wherein the camera is a monocular camera; Fusing the global features of the image to be tested, the local features of the image to be tested, and the depth map features of the environment to be tested to obtain fused features of the image to be tested; Predicting the depth of the obstacle in the image to be tested from the camera according to the fusion features of the image to be tested; The depth map feature of the environment to be tested is obtained based on the ground depth map of the environment to be tested, the ground depth map of the environment to be tested is established based on the depth of at least one point on the ground of the environment to be tested from the camera, and the depth of at least one point on the ground of the environment to be tested from the camera is determined based on the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system, the intrinsic parameters of the camera and the ground equation.

2. The method according to claim 1, before obtaining the fusion features of the image to be tested, further comprising: include: A depth map feature of the environment to be measured is extracted from a ground depth map of the environment to be measured.

3. The method according to claim 2, in, The extracting the depth map features of the environment to be tested from the ground depth map of the environment to be tested includes: The ground depth map of the environment to be tested is input into the backbone network, and the depth map features of the environment to be tested output by the backbone network are obtained.

4. The method according to claim 2, before extracting the depth map features of the environment to be tested from the ground depth map of the environment to be tested, further comprising: include: A ground depth map of the environment to be measured is established according to the depth of at least one point on the ground of the environment to be measured from the camera.

5. The method according to claim 4, before establishing the ground depth map of the environment to be measured, further comprising: include: The depth of at least one point on the ground of the environment to be measured from the camera is determined according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system.

6. The method according to claim 5, in, The step of determining the depth of at least one point on the ground of the environment to be measured from the camera according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system includes: Obtaining coordinates of at least one point on the ground of the environment to be measured in an image coordinate system; The depth of at least one point on the ground of the environment to be measured from the camera is determined according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system, the intrinsic parameters of the camera and the ground equation.

7. The method according to claim 6, in, The step of determining the depth of at least one point on the ground of the environment to be measured from the camera according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system, the intrinsic parameters of the camera and the ground equation comprises: According to the intrinsic parameters of the camera and the ground equation, the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system are reflected and projected to calculate the depth of at least one point on the ground of the environment to be measured from the camera.

8. The method according to any one of claims 1 to 7, in, The step of predicting the depth of the obstacle in the image to be tested from the camera according to the fusion feature of the image to be tested includes: The fusion features of the image to be tested are input into a neural network model, and the depth of the obstacle in the image to be tested output by the neural network model from the camera is obtained.

9. The method according to claim 8, in, The neural network model is a convolutional neural network model or a fully connected neural network model.

10. The method according to any one of claims 1 to 7, in, The image to be tested is a monocular image.

11. An image detection device, include: An acquisition module is used to acquire an image to be tested captured by a camera in the environment to be tested, wherein the camera is a monocular camera; A fusion module, used to fuse the global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested to obtain fusion features of the image to be tested; The depth map feature of the environment to be tested is obtained according to the ground depth map of the environment to be tested, the ground depth map of the environment to be tested is established according to the depth of at least one point on the ground of the environment to be tested from a camera, and the depth of at least one point on the ground of the environment to be tested from the camera is determined according to the coordinates of at least one point on the ground of the environment to be tested in an image coordinate system, the intrinsic parameters of the camera and a ground equation; The prediction module is used to predict the depth of the obstacle in the image to be tested from the camera according to the fusion features of the image to be tested.

12. The device according to claim 11, further comprising: include: The extraction module is used to extract the depth map features of the environment to be tested from the ground depth map of the environment to be tested.

13. The device according to claim 12, in, The extraction module is specifically used to input the ground depth map of the environment to be tested into the backbone network, and obtain the depth map features of the environment to be tested output by the backbone network.

14. The device according to claim 12, further comprising: include: A mapping module is used to establish a ground depth map of the environment to be measured according to the depth of at least one point on the ground of the environment to be measured from the camera.

15. The device according to claim 14, further comprising: include: A calculation module is used to determine the depth of at least one point on the ground of the environment to be measured from the camera according to the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system.

16. The device according to claim 15, in, The calculation module is specifically used to obtain the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system; determine the depth of at least one point on the ground of the environment to be tested from the camera according to the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system, the intrinsic parameters of the camera and the ground equation.

17. The device according to claim 16, in, The calculation module is specifically used to perform reflection projection on the coordinates of at least one point on the ground of the environment to be measured in the image coordinate system according to the intrinsic parameters of the camera and the ground equation, and calculate the depth of at least one point on the ground of the environment to be measured from the camera.

18. The device according to any one of claims 11 to 17, in, The prediction module is specifically used to input the fusion features of the image to be tested into the neural network model, and obtain the depth of the obstacle in the image to be tested from the camera output by the neural network model.

19. The device according to claim 18, in, The neural network model is a convolutional neural network model or a fully connected neural network model.

20. The device according to any one of claims 11 to 17, in, The image to be tested is a monocular image.

21. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-10.

23. A method for image detection, include: The global features of the image to be tested, the local features of the image to be tested and the depth map features of the environment to be tested are fused to obtain the fused features of the image to be tested, wherein the image to be tested is collected by a monocular camera, the depth map features of the environment to be tested are obtained according to the ground depth map of the environment to be tested, the ground depth map of the environment to be tested is established according to the depth of at least one point on the ground of the environment to be tested from the camera, and the depth of at least one point on the ground of the environment to be tested from the camera is determined according to the coordinates of at least one point on the ground of the environment to be tested in the image coordinate system, the intrinsic parameters of the camera and the ground equation; The depth of the obstacle in the image to be tested from the camera is predicted according to the fusion features of the image to be tested.

24. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.