Vehicle perception method, device, electronic device, and machine-readable storage medium

By combining the analysis and conversion of the intermediate results output by the deep learning network and the video image frames, the motion parameters and position information of other vehicles are generated, and the problem of inaccurate perception of rare scenes in the prior art is solved, and the safety of autonomous vehicles is improved sex.

CN112669335BActive Publication Date: 2024-11-05NEUSOFT REACH AUTOMOBILE TECH (SHENYANG) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110115988.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-27
Publication Date
2024-11-05
Estimated Expiration
2041-01-27

AI Technical Summary

Technical Problem

Existing deep learning methods cannot accurately perceive other vehicles or obstacles in rare scenarios in vehicle perception, resulting in limited safety and reliability of autonomous vehicles.

Method used

By analyzing and converting the intermediate results output by the deep learning network, combining continuous video image frames and point cloud diagrams, the motion parameters and position information of the other car are generated, and the Kalman filtering method is used to predict the position of the other car at the next moment, breaking the training sample Bonding, improve perceptual accuracy.

Benefits of technology

It realizes a more accurate perception of the state of other vehicles ahead, ensures the safety and reliability of autonomous vehicles, and expresses vehicle position information through the combination of visual geometric principles and intermediate results, and enhances the perception ability of rare scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112669335B_ABST
    Figure CN112669335B_ABST
Patent Text Reader

Abstract

The present invention provides a vehicle perception method, device, electronic device and machine-readable storage medium, including: obtaining a plurality of continuous video image frames for the current direction of travel of the own vehicle; determining a point cloud image at each moment and motion parameters of another vehicle, motion parameters of the own vehicle and an instance segmentation map corresponding to the point cloud image at each moment according to the plurality of video image frames and a deep learning network model; determining a first conversion map corresponding to the point cloud image at each moment according to the point cloud image at each moment and the corresponding instance segmentation map and motion parameters of the other vehicle; determining a second conversion map corresponding to the point cloud image at each moment according to the first conversion map corresponding to the point cloud image at each moment and the motion parameters of the own vehicle; perceiving and predicting the position information of the other vehicle at the next moment based on the point cloud image corresponding to the second conversion map at each moment, analyzing and converting the intermediate results output by the deep learning network, thereby improving the accuracy of the perception estimation of the state of the other vehicle in front.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle driving, and in particular to a vehicle perception method, device, electronic device and machine-readable storage medium. Background Art

[0002] In the autonomous driving scenario, there will be various other vehicles on the road ahead, and the status of other vehicles will affect the driving decision of the vehicle. In order to ensure driving safety, the autonomous driving vehicle needs to control its own vehicle operations such as deceleration, acceleration, overtaking, or turning, etc., to avoid friction and collision accidents with these other vehicles. Therefore, in order to achieve accurate control of its own vehicle to ensure driving safety, it is very important to perceive other vehicles.

[0003] Among the existing methods, machine learning methods and deep learning methods are generally used for image detection to obtain vehicle perception results. However, such methods are limited by the selection of training samples for the deep learning network model. That is, for business scenarios that are not involved in the training samples, the deep learning method cannot accurately detect and perceive other vehicles, and cannot guarantee the safety and reliability of vehicle driving. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a vehicle perception method, device, electronic device and machine-readable storage medium, which can improve the accuracy of perception and estimation of the state of other vehicles ahead by analyzing and converting the intermediate results output by the deep learning network.

[0005] In a first aspect, an embodiment provides a vehicle perception method, the method comprising:

[0006] Acquire a plurality of continuous video image frames for the current direction of travel of the vehicle;

[0007] According to the multiple video image frames and the deep learning network model, determine a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment, wherein the motion parameters are used to characterize the motion state of the vehicle from a previous moment to a current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment;

[0008] Determine, according to the point cloud image at each moment, the corresponding instance segmentation image and the motion parameters of the other vehicle, a first conversion image corresponding to the point cloud image at each moment, wherein the first conversion image is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment;

[0009] Determine, according to the first conversion graph corresponding to the point cloud graph at each moment and the motion parameters of the vehicle, a second conversion graph corresponding to the point cloud graph at each moment, wherein the second conversion graph is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment;

[0010] Based on the point cloud image at each moment corresponding to the second conversion image, the position information of the other vehicle at the next moment is perceived and predicted.

[0011] In an optional embodiment, the deep learning network model includes a first deep learning network model and a second deep learning network model, and the step of determining a point cloud image at each moment and other vehicle motion parameters, self-vehicle motion parameters, and instance segmentation map corresponding to the point cloud image at each moment according to the multiple video image frames and the deep learning network model includes:

[0012] Determine a point cloud image at each moment according to the front and back adjacent video image frames among the multiple video image frames and in combination with the first deep learning network model and the camera internal parameters;

[0013] The video image frames at each moment are input into the second deep learning network model, and the other vehicle motion parameters, the own vehicle motion parameters and the instance segmentation map corresponding to the video image frames at each moment are output.

[0014] In an optional implementation, the step of determining a point cloud image at each moment according to the front and rear adjacent video image frames among the multiple video image frames and in combination with the first deep learning network model and the camera internal parameters includes:

[0015] Selecting a video image frame at time T and a video image frame at time T-1 of the previous frame from the multiple video image frames, combining the first deep learning network model and the camera internal parameters, obtaining a point cloud image at time T, and repeating the above process until all video image frames at each time are traversed, where T is a natural number greater than 1;

[0016] The step of inputting the video image frame at each moment into the second deep learning network model, and outputting the other vehicle motion parameters, the vehicle motion parameters and the instance segmentation map corresponding to the video image frame at each moment includes:

[0017] The video image frame at time T is input into the second deep learning network model, and the motion parameters of the other vehicle, the motion parameters of the own vehicle and the instance segmentation map corresponding to the video image frame at time T are output. The above process is repeated until the corresponding parameters of the video image frame at each moment are output, and the motion parameters are used to characterize the motion state of the vehicle from time T-1 to time T, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at time T.

[0018] In an optional embodiment, the step of determining the first conversion map corresponding to the point cloud image at each moment according to the point cloud image at each moment and the corresponding instance segmentation map and the motion parameters of the other vehicle includes:

[0019] Determine a point cloud information map within the area where the other vehicle is located in the video image frame at time T according to the point cloud map at time T and the instance segmentation map;

[0020] Based on the point cloud information map in the area where the other vehicle is located in the video image frame at time T and the motion parameters of the other vehicle, a conversion calculation is performed to determine a first conversion map, and the above process is repeated until the first conversion map corresponding to the point cloud map at each moment is determined, and the first conversion map is used to characterize the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at time T.

[0021] In an optional embodiment, the step of perceiving and predicting the position information of the other vehicle at the next moment based on the point cloud image at each moment corresponding to the second conversion image includes:

[0022] Determine a point cloud position information set corresponding to the target vehicle according to the second transformation graph at each moment;

[0023] The point cloud position information set is predicted according to the Kalman filtering method to obtain the point cloud position information of the target vehicle at the next moment.

[0024] In an optional embodiment, the method further comprises:

[0025] Based on the point cloud position information of the target vehicle at the next moment, the current vehicle is controlled to perform corresponding operations.

[0026] In an optional embodiment, the motion parameters include six-degree-of-freedom parameters.

[0027] In a second aspect, an embodiment provides a vehicle sensing device, the device comprising:

[0028] An acquisition module, which acquires a plurality of continuous video image frames for the current direction of travel of the vehicle;

[0029] A first determination module determines, based on the multiple video image frames and the deep learning network model, a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment, wherein the motion parameters are used to characterize the motion state of the vehicle from a previous moment to a current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment;

[0030] a second determination module, determining a first conversion map corresponding to the point cloud map at each moment according to the point cloud map at each moment, the corresponding instance segmentation map and the motion parameters of the other vehicle, wherein the first conversion map is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment;

[0031] a conversion module, determining, according to the first conversion graph corresponding to the point cloud graph at each moment and the motion parameters of the vehicle, a second conversion graph corresponding to the point cloud graph at each moment, wherein the second conversion graph is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment;

[0032] The perception module perceives and predicts the position information of the other vehicle at the next moment based on the point cloud image at each moment corresponding to the second transformation image.

[0033] In a third aspect, an embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any of the aforementioned implementation methods are implemented.

[0034] In a fourth aspect, an embodiment provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of the aforementioned implementations.

[0035] A vehicle perception method and device provided by an embodiment of the present invention obtains a point cloud map at each moment, the motion states of the own vehicle and the other vehicle from the previous moment to the current moment, and the area where the other vehicle is located in the video image frame at each moment through the combination of a deep learning network and continuous video image frames, and determines the intermediate results of the first conversion map and the second conversion map at each moment, wherein the first conversion map is a point cloud map of the area where the other vehicle is located relative to the own vehicle, and then combined with the motion parameters of the own vehicle, obtains the second conversion map for characterizing the actual absolute position of the other vehicle, and based on the point cloud position set of the other vehicle in the second conversion map at each moment, the position information of the other vehicle at the next moment can be perceived, and the intermediate results are combined with the principles of visual geometry to break the constraints of training samples, realize the information expression of the vehicle position through the world coordinate system, and thus improve the accuracy of the perception and estimation of the state of the other vehicle ahead.

[0036] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0037] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a vehicle perception method provided by an embodiment of the present invention;

[0040] Figure 2 A schematic flow chart of another vehicle perception method provided by an embodiment of the present invention;

[0041] Figure 3 A functional module diagram of a vehicle sensing device provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In the early days, machine learning methods were used to detect images and obtain the perception structure of vehicles, but this method was not very accurate. In recent years, in order to further improve the accuracy of vehicle perception results, people have used deep learning methods to perceive vehicles. Vehicle perception results are generally 2D box target detection results, semantic segmentation results, or instance segmentation results. These are only information expressions of the vehicle in the image, and their detection accuracy is limited by the training samples of the deep learning model. If the order of magnitude of the training samples is large and the business scenarios they cover are relatively wide, then the detection accuracy of this deep learning model is high, but the cost is high, which is not conducive to its widespread application in the field of vehicle driving; if the number of training samples is streamlined and the main business scenarios of its main application in vehicle driving are covered, then when some relatively rare scenarios occur in vehicle driving, this deep learning model cannot accurately perceive.

[0045] It is understandable that vehicle driving scenarios are complex and changeable. If the deep learning model cannot detect and identify the situation, other vehicles or obstacles on the road will pose a greater threat to the driving safety of the vehicle. Among them, relatively rare scenarios may include large non-vehicle obstacles in front, such as stones, or other vehicles on the road that violate traffic rules and drive or disappear from the current vehicle's field of vision. At this time, the deep learning model has not been trained with training samples of similar scenarios, and thus cannot accurately perceive obstacles or other vehicles in such rare scenarios, and cannot guarantee the safety and reliability of vehicle driving.

[0046] Based on this, an embodiment of the present invention provides a vehicle perception method, device, electronic device and machine-readable storage medium, which analyzes and converts the intermediate results output by the deep learning network, thereby improving the accuracy of the perception estimation of the state of the vehicle ahead.

[0047] To facilitate understanding of this embodiment, a vehicle perception method disclosed in an embodiment of the present invention is first introduced in detail. This method can be applied to a control processing device, which can be an external device for vehicle perception similar to a driving recorder or a control device integrated in the vehicle, such as a vehicle-like device.

[0048] Figure 1 A flow chart of a vehicle perception method provided in an embodiment of the present invention.

[0049] like Figure 1 As shown, the method comprises the following steps:

[0050] Step S102, acquiring a plurality of continuous video image frames for the current vehicle traveling direction;

[0051] Here, video image frames can be obtained in a variety of ways, and the image frames can be obtained by a collection device installed on the vehicle. As an optional embodiment, the collection device can be a forward-looking camera installed on the front windshield of the vehicle, collecting road videos in the direction of the vehicle's travel to form a plurality of continuous image frames. The forward-looking camera can communicate with the control processing device so that the control processing device obtains the camera internal parameters, or the camera internal parameters can be input in advance according to a manual setting method.

[0052] It can be understood that the self-vehicle is the vehicle driven by the current user, and the other vehicles are other vehicles collected during the driving of the current user's vehicle. Such other vehicles may be driving on the road or parked beside the road.

[0053] Step S104, determining a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment according to the multiple video image frames and the deep learning network model, wherein the motion parameters are used to characterize the motion state of the vehicle from the previous moment to the current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment;

[0054] Among them, the motion parameters include six degrees of freedom parameters. An object has six degrees of freedom in space, namely the freedom of movement along the three rectangular coordinate axes of x, y, and z and the freedom of rotation around these three coordinate axes. The six degrees of freedom movement include: rotation in the three directions of pitch, roll, and yaw, and linear motion in the three directions of longitudinal, lateral, and vertical, which can be used to simulate the driving conditions of a vehicle on the road.

[0055] It should be noted that the deep learning model used to generate and output the other vehicle motion parameters, the vehicle motion parameters and the instance segmentation map corresponding to the point cloud map at each moment combines the deep learning algorithm with the principles of visual geometry.

[0056] Step S106, determining a first conversion map corresponding to the point cloud map at each moment according to the point cloud map at each moment, the corresponding instance segmentation map and the motion parameters of the other vehicle, wherein the first conversion map is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment;

[0057] Step S108, determining a second conversion map corresponding to the point cloud map at each moment according to the first conversion map corresponding to the point cloud map at each moment and the vehicle motion parameters, wherein the second conversion map is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment;

[0058] Among them, since the above-mentioned parameters (such as other vehicle motion parameters, own vehicle motion parameters and instance segmentation map) are generated based on a deep learning model including visual geometry principles, it can be understood that operations such as the first conversion map generated by the above-mentioned instance segmentation map and other vehicle motion parameters, and the conversion process between the first conversion map and the second conversion map realized by the above-mentioned own vehicle motion parameters are all based on visual geometry principles.

[0059] Step S110, based on the point cloud image at each moment corresponding to the second conversion image, perceive and predict the position information of the other vehicle at the next moment.

[0060] In a preferred embodiment of practical application, through the combination of deep learning network and continuous video image frames, the point cloud map at each moment, the motion state of the own vehicle and the other vehicle from the previous moment to the current moment, and the area where the other vehicle is located in the video image frame at each moment are obtained, and the intermediate results of the first conversion map and the second conversion map at each moment are determined, wherein the first conversion map is the point cloud map of the area where the other vehicle is located relative to the own vehicle, and then combined with the motion parameters of the own vehicle, the second conversion map used to characterize the actual absolute position of the other vehicle is obtained, based on the point cloud position set of the other vehicle in the second conversion map at each moment, the position information of the other vehicle at the next moment can be perceived, and the intermediate results are combined with the principles of visual geometry to break the constraints of training samples, realize the information expression of the vehicle position through the world coordinate system, and thus improve the accuracy of the perception and estimation of the state of the other vehicle in front.

[0061] In an optional implementation, the deep learning network model includes a first deep learning network model and a second deep learning network model, and step S104 further includes the following steps:

[0062] Step 1.1), determining a point cloud image at each moment according to the front and rear adjacent video image frames in the multiple video image frames and in combination with the first deep learning network model and the camera internal parameters;

[0063] like Figure 2 As shown, a video image frame at time T and a video image frame at time T-1 of the previous frame are selected from the multiple video image frames, and input into the first deep learning network model to obtain a depth map at time T; combined with the camera internal parameters, a point cloud map at time T is obtained from the depth map at time T, and the above process is repeated until all video image frames at each time are traversed, where T is a natural number greater than 1.

[0064] Step 1.2), input the video image frame at each moment into the second deep learning network model, and output the other vehicle motion parameters, the vehicle motion parameters and the instance segmentation map corresponding to the video image frame at each moment.

[0065] Here, the video image frame at time T is input into the second deep learning network model, and the motion parameters of the other vehicle, the motion parameters of the own vehicle and the instance segmentation map corresponding to the video image frame at time T are output. The above process is repeated until the corresponding parameters of the video image frame at each moment are output. The motion parameters are used to characterize the motion state of the vehicle from time T-1 to time T, and the instance segmentation map is used to segment out the area where the other vehicle is located in the video image frame at time T.

[0066] In an optional implementation manner, step S106 may also be implemented by the following steps, specifically including:

[0067] Step 2.1), determining a point cloud information map within the area where the other vehicle is located in the video image frame at time T according to the point cloud map at time T and the instance segmentation map;

[0068] Step 2.2), based on the point cloud information map in the area where the other vehicle is located in the video image frame at time T and the motion parameters of the other vehicle, a conversion calculation is performed to determine a first conversion map, and the above process is repeated until the first conversion map corresponding to the point cloud map at each moment is determined, and the first conversion map is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at time T.

[0069] Here, if Figure 2 As shown, the first conversion map is jointly determined by the point cloud map at time T, the instance segmentation map and the motion parameters of the other vehicle, and the second conversion map is jointly determined by the first conversion map and the motion parameters of the own vehicle.

[0070] In an optional implementation, step S108 may include the following steps:

[0071] Step 3.1), determining a point cloud position information set corresponding to the target vehicle according to the second transformation graph at each moment;

[0072] Here, the instance segmentation map may segment the area where one other car is located, the area where multiple other cars are located, or the area where no other car is located. Then the second conversion map may also include the situation where one other car, multiple other cars, or no other car is located. As an optional embodiment, if there are multiple other cars at this time, the second conversion map at each moment is used to collect point cloud location information according to the category of other cars to which they belong. For example, the second conversion map includes the area where the point clouds of other cars A, B, and C are located. The point cloud location information of other car A at each moment is collected in one place and recorded as Available The same goes for his car B and his car C.

[0073] Step 3.2), filtering the point cloud position information set according to the Kalman filtering method, and then realizing prediction to obtain the point cloud position information of the target vehicle at the next moment.

[0074] In an optional embodiment, the method further comprises:

[0075] Step 4.1), based on the point cloud position information of the target vehicle at the next moment, control the current vehicle to perform corresponding operations, such as controlling the vehicle to stop and avoid, or detecting the distance to the target vehicle, controlling the speed of the vehicle, etc., to ensure the reliability of vehicle driving.

[0076] like Figure 3 As shown, the embodiment of the present invention further provides a vehicle sensing device 200, the device comprising:

[0077] An acquisition module 201 acquires a plurality of continuous video image frames for the current direction of travel of the vehicle;

[0078] The first determination module 202 determines, based on the multiple video image frames and the deep learning network model, a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment, wherein the motion parameters are used to characterize the motion state of the vehicle from the previous moment to the current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment;

[0079] The second determination module 203 determines a first conversion map corresponding to the point cloud map at each moment according to the point cloud map at each moment, the corresponding instance segmentation map and the motion parameters of the other vehicle, wherein the first conversion map is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment;

[0080] The conversion module 204 determines, based on the first conversion graph corresponding to the point cloud graph at each moment and the vehicle motion parameters, that the point cloud graph at each moment corresponds to a second conversion graph, wherein the second conversion graph is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment;

[0081] The perception module 205 perceives and predicts the position information of the other vehicle at the next moment based on the point cloud image at each moment corresponding to the second conversion image.

[0082] The embodiment of the present invention obtains the first conversion map of the perception intermediate result through a deep learning model and a point cloud map, determines the second conversion map through the principle of visual geometry, completes the detection part, and then perceives the point cloud position information set of the other vehicle in the second conversion map at each moment to predict the position information of the other vehicle at the next moment, and then controls the vehicle to perform corresponding operations to ensure the safety of the self-driving car.

[0083] Figure 4 Schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. Figure 4 As shown, the electronic device 300 includes: a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304 and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303 and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the vehicle perception method described in the above embodiment by reading and executing the machine-executable instructions for vehicle perception in the machine-readable storage medium 301.

[0084] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0085] The non-volatile medium may be a non-volatile memory, a flash memory, a storage drive (such as a hard drive), any type of storage disk (such as a CD, DVD, etc.), or a similar non-volatile storage medium, or a combination thereof.

[0086] It can be understood that the specific operation methods of each functional module in this embodiment can refer to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0087] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program. When the computer program code is executed, the vehicle perception method described in any of the above embodiments can be implemented. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0089] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0090] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0091] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed by the present invention, or can easily conceive of changes, or make equivalent replacements for some of the technical features therein. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention.

Claims

1. A vehicle perception method, characterized in that: The method comprises: Acquire a plurality of continuous video image frames for the current direction of travel of the vehicle; According to the multiple video image frames and the deep learning network model, determine a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment, wherein the motion parameters are used to characterize the motion state of the vehicle from a previous moment to a current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment; Determine, according to the point cloud image at each moment, the corresponding instance segmentation image and the motion parameters of the other vehicle, a first conversion image corresponding to the point cloud image at each moment, wherein the first conversion image is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment; Determine, according to the first conversion graph corresponding to the point cloud graph at each moment and the motion parameters of the vehicle, a second conversion graph corresponding to the point cloud graph at each moment, wherein the second conversion graph is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment; Based on the point cloud image at each moment corresponding to the second conversion image, the position information of the other vehicle at the next moment is perceived and predicted; The deep learning network model includes a first deep learning network model and a second deep learning network model. The step of determining a point cloud image at each moment and other vehicle motion parameters, self-vehicle motion parameters and an instance segmentation map corresponding to the point cloud image at each moment according to the multiple video image frames and the deep learning network model includes: Determine a point cloud image at each moment according to the front and back adjacent video image frames among the multiple video image frames and in combination with the first deep learning network model and the camera internal parameters; Input the point cloud image at each moment into a second deep learning network model, and output the other vehicle motion parameters, the vehicle motion parameters, and the instance segmentation map corresponding to the point cloud image at each moment; The motion parameters include six-degree-of-freedom parameters.

2. The method according to claim 1, characterized in that The step of determining a point cloud image at each moment according to the front and rear adjacent video image frames in the plurality of video image frames and in combination with the first deep learning network model and the camera internal parameters includes: Selecting a video image frame at time T and a video image frame at time T-1 of the previous frame from the multiple video image frames, combining the first deep learning network model and the camera internal parameters, obtaining a point cloud image at time T, and repeating the above process until all video image frames at each time are traversed, where T is a natural number greater than 1; The step of inputting the point cloud image at each moment into a second deep learning network model, and outputting the other vehicle motion parameters, the vehicle motion parameters, and the instance segmentation map corresponding to the point cloud image at each moment includes: The point cloud image at time T is input into the second deep learning network model, and the motion parameters of the other vehicle, the motion parameters of the own vehicle and the instance segmentation map corresponding to the point cloud image at time T are output. The above process is repeated until the corresponding parameters of the point cloud image at each moment are output. The motion parameters are used to characterize the motion state of the vehicle from time T-1 to time T, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at time T.

3. The method according to claim 2, characterized in that The step of determining a first conversion map corresponding to the point cloud image at each moment according to the point cloud image at each moment and the corresponding instance segmentation map and the motion parameters of the other vehicle comprises: Determine a point cloud information map within the area where the other vehicle is located in the video image frame at time T according to the point cloud map at time T and the instance segmentation map; Based on the point cloud information map in the area where the other vehicle is located in the video image frame at time T and the motion parameters of the other vehicle, a conversion calculation is performed to determine a first conversion map, and the above process is repeated until the first conversion map corresponding to the point cloud map at each moment is determined, and the first conversion map is used to characterize the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at time T.

4. The method according to claim 1, characterized in that: The step of perceiving and predicting the position information of the other vehicle at the next moment based on the point cloud image at each moment corresponding to the second conversion image comprises: Determine a point cloud position information set corresponding to the target vehicle according to the second transformation graph at each moment; The point cloud position information set is predicted according to the Kalman filtering method to obtain the point cloud position information of the target vehicle at the next moment.

5. The method according to claim 4, characterized in that The method further comprises: Based on the point cloud position information of the target vehicle at the next moment, the current vehicle is controlled to perform corresponding operations.

6. A vehicle sensing device, characterized in that: The device comprises: An acquisition module, which acquires a plurality of continuous video image frames for the current direction of travel of the vehicle; A first determination module determines, based on the multiple video image frames and the deep learning network model, a point cloud image at each moment and motion parameters of the other vehicle, motion parameters of the own vehicle, and an instance segmentation map corresponding to the point cloud image at each moment, wherein the motion parameters are used to characterize the motion state of the vehicle from a previous moment to a current moment, and the instance segmentation map is used to segment the area where the other vehicle is located in the video image frame at each moment; a second determination module, determining a first conversion map corresponding to the point cloud map at each moment according to the point cloud map at each moment, the corresponding instance segmentation map and the motion parameters of the other vehicle, wherein the first conversion map is used to represent the position of the point cloud information in the area where the other vehicle is located relative to the own vehicle at the current moment; a conversion module, determining, according to the first conversion graph corresponding to the point cloud graph at each moment and the motion parameters of the vehicle, a second conversion graph corresponding to the point cloud graph at each moment, wherein the second conversion graph is used to represent the actual absolute position of the point cloud information in the area where the other vehicle is located at the current moment; A perception module, which perceives and predicts the position information of the other vehicle at the next moment based on the point cloud image at each moment corresponding to the second conversion image; The deep learning network model includes a first deep learning network model and a second deep learning network model, and the first determination module is further used to determine the point cloud image at each moment according to the front and rear adjacent video image frames in the multiple video image frames, in combination with the first deep learning network model and the camera internal parameters; input the point cloud image at each moment into the second deep learning network model, and output the other vehicle motion parameters, the own vehicle motion parameters and the instance segmentation map corresponding to the point cloud image at each moment; The motion parameters include six-degree-of-freedom parameters.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A machine-readable storage medium, characterized in that:The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of claims 1 to 5.