Motor vehicle obstacle detection method and device and computer readable storage medium

Abstract

The embodiment of the invention provides a motor vehicle obstacle detection method and device and a computer readable storage medium, and the method comprises the steps: extracting image frames frame by frame from a video image around a motor vehicle collected and transmitted by a vehicle-mounted camera, and screening a candidate region containing an obstacle from each image frame; acquiring a ray projection image of the candidate area of each image frame based on a ray projection mechanism of a camera imaging model; all the ray projection images are input into a feature extraction network to extract ray projection image features, and motion compensation alignment is performed on the ray projection image corresponding to the currently processed image frame and then the ray projection image is input into the feature extraction network from a second image frame in the time sequence; fusing each ray projection image feature according to a time sequence and based on motion compensation to obtain a fusion feature; and inputting the fusion features into an obstacle boundary and perigee detection model to identify the obstacle. According to the embodiment, the obstacle detection and recognition precision can be effectively improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of motor vehicle obstacle detection, and more particularly to a motor vehicle obstacle detection method, device, and computer-readable storage medium. Background Art

[0002] The motor vehicle assisted driving system is usually equipped with a motor vehicle obstacle detection device based on image vision, so as to monitor obstacles appearing around the motor vehicle in real time during the driving process of the motor vehicle, and to provide early warning and / or take obstacle avoidance measures.

[0003] An existing motor vehicle obstacle detection method based on image vision mainly includes the following steps: first, a predetermined obstacle recognition algorithm is used to screen out candidate frames containing obstacles around the motor vehicle from the image frames taken by the vehicle-mounted camera, and then a predetermined feature extraction network model is used to extract image features from the candidate frames, and finally a deep learning-based network model is used to classify and identify the image features to determine the type of obstacle.

[0004] However, the inventors found during the specific implementation that the above method only relies on the image features in a single image frame to identify obstacles, the obstacle identification accuracy is poor, and the spatial perception accuracy of the obstacles is low. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a motor vehicle obstacle detection method that can effectively improve obstacle detection and recognition accuracy.

[0006] A further technical problem to be solved by the embodiments of the present invention is to provide a motor vehicle obstacle detection device that can effectively improve obstacle detection and recognition accuracy.

[0007] A further technical problem to be solved by the embodiments of the present invention is to provide a computer-readable storage medium capable of storing a computer program that can effectively improve obstacle detection and recognition accuracy.

[0008] In order to solve the above technical problems, the embodiment of the present invention first provides the following technical solution: a motor vehicle obstacle detection method, comprising the following steps: Extracting image frames frame by frame from the video images of the surrounding environment of the motor vehicle collected and transmitted by the on-board camera, and screening out candidate areas containing obstacles in each image frame; Acquire a ray projection image of the candidate area of ​​each image frame based on a ray projection mechanism of a camera imaging model; Inputting all ray projection images into a feature extraction network to extract features of the ray projection images, wherein, starting from the second image frame in the time sequence, the ray projection images corresponding to the currently processed image frame are firstly motion-compensated aligned and then input into the feature extraction network; fusing the features of each ray projection image in time sequence and based on motion compensation to obtain a fusion feature; and The fused features are input into the obstacle boundary and perigee detection model to identify the obstacle.

[0009] Furthermore, the ray projection mechanism based on the camera imaging model to obtain the ray projection image of the candidate area of ​​each image frame specifically includes: Determine, based on the camera extrinsic parameters of the vehicle-mounted camera, in the image frame, a pixel coordinate point where the camera center point of the vehicle-mounted camera is projected onto the road surface on which the motor vehicle is traveling; Divide the candidate region into a number of sub-regions; Constructing target rays connecting the pixel coordinate points with the pixel center points of each of the sub-areas respectively; and A ray projection image is constructed based on each target ray in the image frame, and the RGB values ​​of each row of pixels in the ray projection image correspond to the RGB values ​​of each pixel on a target ray according to the arrangement order of each target ray.

[0010] Furthermore, performing motion compensation alignment on the ray projection image corresponding to the currently processed image frame and then inputting the ray projection image into the feature extraction network specifically includes: Projecting each target ray in the currently processed image frame to the historical image frame based on the inter-frame motion vector of the vehicle-mounted camera to generate a corresponding historical target ray in the historical image frame; Constructing a historical projection image based on each historical target ray in the historical image frame; and The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

[0011] Furthermore, the fusing of the ray projection image features in time sequence and based on motion compensation to obtain the fused features specifically refers to: fusing the historical image features and the current image features extracted by the feature extraction network from the historical projection images and the ray projection images corresponding to the currently processed image frame respectively to obtain the fused features.

[0012] Furthermore, the historical image features and the current image features are added and fused to form the fused features.

[0013] Furthermore, after constructing the ray projection image and the historical projection image, the RGB values ​​of the missing pixels in each projection image are first completed based on the proximity principle of pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction.

[0014] Furthermore, the obstacle boundary and perigee detection model is a three-layer MLP network structure model, and the fusion features are input into the three-layer MLP network structure model to identify the projection point of the obstacle on the driving road and the type of the obstacle.

[0015] Furthermore, a generalized obstacle detection algorithm model is used to filter out candidate areas containing obstacles around the motor vehicle from the image frame.

[0016] On the other hand, in order to solve the above-mentioned further technical problems, an embodiment of the present invention further provides the following technical solutions: a motor vehicle obstacle detection device connected to a vehicle-mounted camera, the device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the motor vehicle obstacle detection method as described in any one of the above-mentioned items when executing the computer program.

[0017] On the other hand, in order to solve the above-mentioned further technical problems, an embodiment of the present invention further provides the following technical solution: a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method as described in any one of the above-mentioned items.

[0018] After adopting the above technical scheme, the embodiment of the present invention has at least the following beneficial effects: after screening out the candidate area containing the obstacle in the image frame, the embodiment of the present invention first obtains the ray projection image of the candidate area of ​​each image frame based on the ray projection mechanism of the camera imaging model, and then inputs all the ray projection images into the feature extraction network to extract the ray projection image features, and starting from the second image frame in the time sequence, for the image frame currently being processed, the corresponding ray projection image will be first motion compensated and aligned before being input into the feature extraction network, which can improve the accuracy of the image features of the image frame currently being processed; after further fusing the features of each ray projection image according to the time sequence and based on motion compensation to obtain the fused features, the fused features are input into the preset corresponding obstacle boundary and perigee detection model to identify the obstacle, which can effectively improve the detection and recognition accuracy of the obstacle, and is conducive to improving the efficiency and success rate of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1The following is a flowchart of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0020] Figure 2 The figure is a schematic diagram of an image frame of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0021] Figure 3 This is a specific flow chart of step S2 of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0022] Figure 4 The following is a principle block diagram of an optional embodiment of the motor vehicle obstacle detection device of the present invention.

[0023] Figure 5 This is a functional module diagram of an optional embodiment of the motor vehicle obstacle detection device of the present invention. DETAILED DESCRIPTION

[0024] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments and features in the embodiments of the present application may be combined with each other without conflict.

[0025] like Figure 1 As shown, an optional embodiment of the present invention provides a motor vehicle obstacle detection method, comprising the following steps: S1: extracting image frames frame by frame from the video image of the surrounding environment of the motor vehicle collected and transmitted by the vehicle-mounted camera 1, and screening out candidate areas containing obstacles in each image frame; S2: acquiring a ray projection image of the candidate area of ​​each image frame based on a ray projection mechanism of a camera imaging model; S3: inputting all ray projection images into a feature extraction network to extract features of the ray projection images, wherein, starting from the second image frame in the time sequence, the ray projection images corresponding to the currently processed image frame are firstly motion-compensated aligned and then input into the feature extraction network; S4: fusing the features of each ray projection image in time sequence and based on motion compensation to obtain a fusion feature; and S5: Input the fused features into the obstacle boundary and perigee detection model to identify the obstacle.

[0026] The embodiment of the present invention obtains the ray projection image of the candidate area of ​​each image frame based on the ray projection mechanism of the camera imaging model after screening out the candidate area containing the obstacle in the image frame, and then inputs all the ray projection images into the feature extraction network to extract the ray projection image features, and starting from the second image frame in the time sequence, for the image frame currently being processed, the corresponding ray projection image will be firstly motion-compensated and aligned before being input into the feature extraction network, which can improve the accuracy of the image features of the image frame currently being processed; further, after the ray projection image features are fused in time sequence and based on motion compensation to obtain the fused features, the fused features are input into the preset corresponding obstacle boundary and perigee detection model to identify the obstacle, which can effectively improve the detection and recognition accuracy of the obstacle, and is conducive to improving the efficiency and success rate of obstacle detection.

[0027] In the specific implementation, in step S4, the backbone network of the feature extraction network adopts the ResNet-34 network model. Of course, in order to facilitate the addition and fusion of the extracted features, a 1×1 convolutional layer can be connected after the ResNet-34 network model to convert the feature size output by the ResNet-34 network model into a specific size (for example: 3×254×254). In an optional embodiment of the present invention, Figure 2 and Figure 3 As shown, the step S2 specifically includes: S21: determining, in the image frame based on the camera extrinsic parameters of the vehicle-mounted camera 1, a pixel coordinate point where the camera center point of the vehicle-mounted camera 1 is projected onto the road surface on which the motor vehicle is traveling; S22: Divide the candidate region into a number of sub-regions; S23: constructing target rays connecting the pixel coordinate points with the pixel center points of each sub-area respectively; and S24: constructing a ray projection image based on each target ray in the image frame, wherein the RGB value of each row of pixels in the ray projection image corresponds to the RGB value of each pixel on a target ray according to the arrangement order of the target rays.

[0028] In this embodiment, the pixel coordinates of the camera center point of the vehicle-mounted camera 1 in the image frame projected on the road surface of the motor vehicle are first determined, and then the candidate area is equally divided into a number of sub-areas, and the pixel coordinates and the pixel center points of each sub-area are sequentially connected to construct a plurality of target rays, and each target ray in the image frame is sequentially corresponding to a row of pixel points of the ray projection image, so that a ray projection image can be finally formed. In a specific implementation, the RGB values ​​of each row of pixel points of the ray projection image correspond to each target ray in a clockwise arrangement order.

[0029] See also Figure 2 A specific application example is shown, the candidate area is R_seed, that is, the pedestrian external frame in the figure, and the candidate area is equally divided into multiple sub-areas of 5×5 resolution size in the figure, and the center point of each sub-area is represented by (x_c, y_c); the pixel coordinate point of the camera center point of the on-board camera calculated in the current image frame is projected on the driving road of the motor vehicle as (X0_bev, Y0_bev), and the target ray is represented by Line_raw; after specific experimental measurements, it is found that the spatial perception accuracy of the motor vehicle obstacle detection method provided by the embodiment of the present invention is less than 20cm.

[0030] In an optional embodiment of the present invention, performing motion compensation alignment on the ray projection image corresponding to the currently processed image frame and then inputting the ray projection image into the feature extraction network specifically includes: Projecting each target ray in the currently processed image frame to the historical image frame based on the inter-frame motion vector of the vehicle-mounted camera 1 to generate a corresponding historical target ray in the historical image frame; Constructing a historical projection image based on each historical target ray in the historical image frame; and The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

[0031] In this embodiment, after projecting each target ray in the currently processed image frame to the historical image frame based on the inter-frame motion vector of the vehicle-mounted camera 1, the same method is used to construct the historical projection image based on each historical target ray in the historical image frame, and finally the historical projection image and the ray projection image corresponding to the currently processed image frame are respectively input into the feature extraction network, so that the motion compensation alignment of the ray projection image corresponding to the currently processed image frame can be achieved. In specific implementation, the historical image frame can be the previous frame of the currently processed image frame or the previous multiple frames, which needs to be determined in combination with the computing power cost and detection efficiency.

[0032] In an optional embodiment of the present invention, the step S4 specifically refers to: fusing the historical image features and the current image features extracted by the feature extraction network from the historical projection image and the ray projection image corresponding to the currently processed image frame to obtain a fused feature. In this embodiment, the historical image features and the current image features extracted by the feature extraction network from the historical projection image and the ray projection image corresponding to the currently processed image frame are used to form a fused feature by fusing the historical image features and the current image features, thereby improving the robustness of the image features and realizing motion compensation of the current image features.

[0033] In an optional embodiment of the present invention, the historical image features and the current image features are added and fused to form the fused features. In this embodiment, image feature fusion is achieved by feature addition, and the fusion method is simple and has high data processing efficiency.

[0034] In an optional embodiment of the present invention, after constructing the ray projection image and the historical projection image, the RGB values ​​of the missing pixels in each projection image are first supplemented based on the proximity principle of the pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction. In this embodiment, since the number of pixels on different target rays in the same image frame is different, after the projection image is constructed based on the target rays in the same image frame, the number of valid pixels in each row of the projection image is different, which will cause some rows of the projection image to have missing pixels. Therefore, in order to improve the comprehensiveness of subsequent image feature extraction and avoid feature extraction errors, the projection image is processed by supplementing the pixels in the same row based on the proximity principle, that is, the RGB values ​​of the valid pixels in the same row adjacent to the missing pixels are supplemented with the RGB values ​​of the missing pixels.

[0035] In an optional embodiment of the present invention, the obstacle boundary and perigee detection model is a three-layer MLP (Multi-Layer Perceptron) network structure model, and the fusion feature is input into the three-layer MLP network structure model to identify the projection point of the obstacle on the road surface and the type of the obstacle. In this embodiment, a three-layer MLP network structure model is used as a detection head to detect, identify and predict the fusion feature, and the recognition accuracy is high; it can be understood that for obstacles in contact with the road surface, the projection point of the obstacle on the road surface is the contact point, and for suspended obstacles, the projection point of the obstacle on the road surface is the intersection of the projection of the suspended obstacle to the road surface; in addition, the type of obstacle is the classification of obstacles, such as pedestrians, trees, vehicles, etc.

[0036] In an optional embodiment of the present invention, a generalized obstacle detection algorithm model is used to filter out candidate areas containing obstacles around the motor vehicle from the image frame. In this embodiment, the generalized obstacle detection algorithm model is a common algorithm that can realize obstacle detection without target detection or semantic segmentation, for example: Patent Publication No. CN116092050 A, entitled "Motor Vehicle Obstacle Detection Method" invention patent application, which will not be repeated here.

[0037] In the present invention, the vehicle-mounted camera 1 is a fisheye camera, and the fisheye image outputted by it, due to image distortion, each target ray in the distorted fisheye image should be arranged in a fan shape with the camera center as the origin, and the image frame is an image after distortion correction. Based on this, the target rays within the field of view of -60 degrees to +60 degrees can be sequentially constructed in a clockwise arrangement order to form a projection image. Correspondingly, the three-layer MLP network structure model can also set the output to 120 perception interfaces, and the 120 perception interfaces correspond to a horizontal viewing angle range of 60 degrees on the left and right with the camera center line as 0 degrees.

[0038] On the other hand, Figure 4 As shown, an optional embodiment of the present invention provides a motor vehicle obstacle detection device 3, which is connected to a vehicle-mounted camera 1. The device 3 includes a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 30. When the processor 30 executes the computer program, the motor vehicle obstacle detection method as described in any one of the above items is implemented.

[0039] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the motor vehicle obstacle detection device 3. For example, the computer program may be divided into Figure 5 The functional modules in the motor vehicle obstacle detection device 3, wherein the image extraction and area screening module 41, the ray projection image construction module 42, the image compensation module 43, the feature fusion module 44 and the obstacle recognition module 45 respectively execute the above steps S1 to S5.

[0040] The motor vehicle obstacle detection device 3 may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The motor vehicle obstacle detection device 3 may include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art may understand that the schematic diagram is only an example of the motor vehicle obstacle detection device 3 and does not constitute a limitation on the motor vehicle obstacle detection device 3. The motor vehicle obstacle detection device 3 may include more or less components than shown in the figure, or a combination of certain components, or different components. For example, the motor vehicle obstacle detection device 3 may also include input and output devices, network access devices, buses, etc.

[0041] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 30 is the control center of the motor vehicle obstacle detection device 3, and uses various interfaces and lines to connect various parts of the entire motor vehicle obstacle detection device 3.

[0042] The memory 32 can be used to store the computer program and / or module. The processor 30 realizes various functions of the motor vehicle obstacle detection device 3 by running or executing the computer program and / or module stored in the memory 32 and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a graphic recognition function, a graphic overlay function, etc.), etc.; the data storage area can store data (such as graphic data, etc.) created according to the use of the control device, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0043] If the functions described in the embodiments of the present invention are implemented in the form of software function modules or units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0044] On the other hand, an optional embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method as described in any one of the above.

[0045] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0046] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.

Claims

1. A motor vehicle obstacle detection method, characterized in that: The method comprises the following steps: Extracting image frames frame by frame from the video images of the surrounding environment of the motor vehicle collected and transmitted by the on-board camera, and screening out candidate areas containing obstacles in each image frame; Acquire a ray projection image of the candidate area of ​​each image frame based on a ray projection mechanism of a camera imaging model; Inputting all ray projection images into a feature extraction network to extract features of the ray projection images, wherein, starting from the second image frame in the time sequence, the ray projection images corresponding to the currently processed image frame are firstly motion-compensated aligned and then input into the feature extraction network; fusing the features of each ray projection image in time sequence and based on motion compensation to obtain a fusion feature; and The fused features are input into the obstacle boundary and perigee detection model to identify the obstacle.

2. The motor vehicle obstacle detection method according to claim 1, characterized in that: The method of obtaining the ray projection image of the candidate area of ​​each image frame based on the ray projection mechanism of the camera imaging model specifically includes: Determine, based on the camera extrinsic parameters of the vehicle-mounted camera, in the image frame, a pixel coordinate point where the camera center point of the vehicle-mounted camera is projected onto the road surface on which the motor vehicle is traveling; Divide the candidate region into a number of sub-regions; Constructing target rays connecting the pixel coordinate points with the pixel center points of each of the sub-areas respectively; and A ray projection image is constructed based on each target ray in the image frame, and the RGB values ​​of each row of pixels in the ray projection image correspond to the RGB values ​​of each pixel on a target ray according to the arrangement order of each target ray.

3. The motor vehicle obstacle detection method according to claim 1 or 2, characterized in that: The step of performing motion compensation alignment on the ray projection image corresponding to the currently processed image frame and then inputting the ray projection image into the feature extraction network specifically includes: Projecting each target ray in the currently processed image frame to the historical image frame based on the inter-frame motion vector of the vehicle-mounted camera to generate a corresponding historical target ray in the historical image frame; Constructing a historical projection image based on each historical target ray in the historical image frame; and The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

4. The motor vehicle obstacle detection method according to claim 3, characterized in that: The fusing of the ray projection image features in time sequence and based on motion compensation to obtain the fused features specifically refers to: fusing the historical image features and the current image features extracted by the feature extraction network from the historical projection images and the ray projection images corresponding to the currently processed image frame respectively to obtain the fused features.

5. The motor vehicle obstacle detection method according to claim 4, characterized in that: The historical image features and the current image features are added and fused to form the fused features.

6. The motor vehicle obstacle detection method according to claim 3, characterized in that: After constructing the ray projection image and the historical projection image, the RGB values ​​of the missing pixels in each projection image are first completed based on the proximity principle of pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction.

7. The motor vehicle obstacle detection method according to claim 1, characterized in that: The obstacle boundary and perigee detection model is a three-layer MLP network structure model, and the fusion features are input into the three-layer MLP network structure model to identify the projection point of the obstacle on the driving road surface and the type of the obstacle.

8. The motor vehicle obstacle detection method according to claim 1, characterized in that: A generalized obstacle detection algorithm model is used to screen out candidate areas containing obstacles in the image frame.

9. A motor vehicle obstacle detection device connected to a vehicle-mounted camera, characterized in that: The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the motor vehicle obstacle detection method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method according to any one of claims 1 to 8.

Images