Vehicle chassis collision early warning method and device, computer equipment and storage medium

Abstract

The invention relates to a vehicle chassis collision early warning method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining target images in front of a vehicle at different virtual view angles based on an image obtaining device of the vehicle; inputting the target images under different virtual view angles into a deep neural network model for reasoning to obtain target prediction information of a target obstacle in an original image of the target images; according to the target prediction information and the obtained vehicle state information, generating early warning information of the target obstacle at the target position; and performing early warning operation based on the early warning information. Through the method and the device, the problem that detection can only be realized after collision and is not predictive in the prior art is solved, and the target prediction information of the target obstacle is accurately detected in advance, so that early warning operation is performed in advance, occurrence of collision or rubbing is reduced, and driving safety is improved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a vehicle chassis collision warning method, device, computer equipment and storage medium. Background Art

[0002] With the development of the automobile industry, additional requirements are put forward for automobiles in addition to basic driving functions. Among them, chassis contact warning requirements are one of the additional requirements. The current implementation scheme is: using sensors installed on the chassis to generate electrical signals through collisions, send them to the battery management domain controller, and then give warning information to the driver. However, the disadvantage of this scheme is that it can only detect after a collision occurs and is not predictive.

[0003] There is currently no effective solution to the problem that related technologies can only detect after a collision occurs and are not predictive. Summary of the invention

[0004] In this embodiment, a vehicle chassis collision warning method, device, computer equipment and storage medium are provided to solve the problem that the related technology can only detect after the collision occurs and has no predictive ability.

[0005] In a first aspect, a vehicle chassis collision warning method is provided in this embodiment, comprising:

[0006] Based on the image acquisition device of the vehicle, a target image under different virtual viewing angles in front of the vehicle is acquired;

[0007] Inputting the target images under different virtual perspectives into a deep neural network model for inference to obtain target prediction information of target obstacles in the original image of the target image;

[0008] According to the target prediction information and the acquired vehicle status information, early warning information of the target obstacle at the target position is generated; and an early warning operation is performed based on the early warning information.

[0009] In some embodiments, the image acquisition device on the vehicle acquires the target image under different virtual perspectives of the vehicle, including:

[0010] Based on an image acquisition device on a vehicle, an original image in the target image of the vehicle and an internal parameter and an external parameter of the image acquisition device are acquired;

[0011] The original image is calculated according to the internal parameters and the external parameters to obtain a depth map and a columnar expansion map in the target image.

[0012] In some embodiments, inputting the target images under different virtual perspectives into a deep neural network model for inference to obtain target prediction information of target obstacles in the original image of the target image includes:

[0013] Inputting the original image and the depth image of the target image into the first network model of the deep neural network model for inference to obtain a corresponding feature map; the feature map includes first prediction information of the obstacle;

[0014] Inputting the feature map and the columnar expansion map of the target image into a second network model of the deep neural network model for inference to obtain second prediction information of the obstacle;

[0015] A post-processing module based on the deep neural network model performs fusion judgment according to the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image.

[0016] In some of the embodiments, the first network model includes a first convolution module and multiple layers of codecs; each layer of the codec includes an encoder and a decoder; the first convolution module serves as the top layer; the output of the encoder of this layer serves as the input of the encoder of the next layer and the decoder of this layer; the output of the decoder of this layer serves as the input of the decoder of the next layer.

[0017] In some of the embodiments, the second network model includes a second convolution module, a pooling module, and a fully connected layer;

[0018] The second convolution module is connected to the pooling module and is used to extract features from the feature map and the columnar expansion map to obtain a feature sequence;

[0019] The pooling module is connected to the fully connected layer and is used to perform average pooling on the feature sequence to obtain a feature vector;

[0020] The fully connected layer is used to transform the feature vector to obtain the second prediction information of the obstacle.

[0021] In some embodiments, the performing fusion judgment according to the first prediction information and the second prediction information to obtain the target prediction information of the target obstacle in the original image includes:

[0022] Compare the first prediction information with the second prediction information to determine whether the obstacle is at a local extreme point in the depth map;

[0023] If the obstacle is not located at a local extreme point in the depth map, the obstacle is discarded;

[0024] If the obstacle is located at a local extreme point in the depth map, the obstacle is used as the target obstacle, and corresponding target prediction information is output.

[0025] In some embodiments, the performing the warning operation based on the warning information includes:

[0026] Based on the warning information, controlling the screen of the vehicle to display an alarm icon, prompt or warning light;

[0027] or, based on the warning information, controlling the horn of the vehicle to play a prompt message;

[0028] Or, based on the warning information, raising the suspension height of the vehicle.

[0029] In a second aspect, a vehicle chassis collision warning device is provided in this embodiment, including: an acquisition module, a reasoning module and a warning module;

[0030] The acquisition module is used to acquire target images in different virtual perspectives in front of the vehicle based on the image acquisition device of the vehicle;

[0031] The inference module is used to input the target image under different virtual perspectives into a deep neural network model for inference, so as to obtain target prediction information of the target obstacle in the original image of the target image;

[0032] The warning module is used to generate warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle status information; and perform a warning operation based on the warning information.

[0033] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the vehicle chassis collision warning method described in the first aspect when executing the computer program.

[0034] In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the vehicle chassis collision warning method described in the first aspect is implemented.

[0035] Compared with the related art, the vehicle chassis collision warning method, device, computer equipment and storage medium provided in this embodiment obtain target images under different virtual perspectives in front of the vehicle through a vehicle-based image acquisition device; input the target images under different virtual perspectives into a deep neural network model for inference to obtain target prediction information of the target obstacle in the original image of the target image; generate warning information of the target obstacle at the target position based on the target prediction information and the obtained vehicle status information; and perform warning operations based on the warning information, which solves the problem that the related art can only realize detection after the collision occurs and has no predictiveness, and accurately detects the target prediction information of the target obstacle in advance, so as to perform warning operations early, reduce the occurrence of collisions or scratches, and improve driving safety.

[0036] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 This is a hardware structure block diagram of a 360-degree surround view system provided in one embodiment of the present application;

[0039] Figure 2 This is a flow chart of a vehicle chassis collision warning method provided by an embodiment of the present application;

[0040] Figure 3 is a flowchart of step S210;

[0041] Figure 4 It is a schematic diagram of the structure of a deep neural network model provided by an embodiment of the present application;

[0042] Figure 5 is a structural diagram of a first network model provided in an embodiment of the present application;

[0043] Figure 6 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0044] Figure 7 is a schematic diagram of the structure of a first convolution module provided in an embodiment of the present application;

[0045] Figure 8 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0046] Fig. 9It is a schematic diagram of the structure of a fully connected conditional random field provided by an embodiment of the present application;

[0047] Fig.10 is a schematic diagram of the calculation relationship between small areas provided in an embodiment of the present application;

[0048] Fig.11 is a structural diagram of a unary operation unit provided in one embodiment of the present application;

[0049] Fig.12 is a schematic diagram of the network structure of a second network model provided in an embodiment of the present application;

[0050] Fig.13 It is a structural block diagram of a vehicle chassis collision warning device provided in one embodiment of the present application.

[0051] In the figure: 10, image acquisition device; 20, domain controller; 30, alarm; 210, acquisition module; 220, reasoning module; 230, early warning module. DETAILED DESCRIPTION

[0052] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiment provided in this embodiment can be executed in the domain controller 20, terminal, computer or similar computing device in the 360-degree surround view system. Figure 1 As shown, the 360-degree surround view system may include an image acquisition device 10, a domain controller 20, and an alarm 30 connected in sequence. The image acquisition device 10 is used to collect target images in different virtual perspectives in front of the vehicle. The domain controller 20 may be an autonomous driving domain controller or a cockpit domain controller, etc., for running various method embodiments; the alarm 30 may be a central control screen, a horn, a warning light, a vibrator in a seat, etc., for performing specific alarm operations.

[0055] For example, the method embodiment is run on the domain controller 20, which may include one or more processors and a memory for storing data, wherein the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above-mentioned domain controller 20 may also include a transmission device and an input and output device for communication functions. It can be understood by those skilled in the art that the domain controller 20 may also include more or fewer components.

[0056] The memory can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the vehicle chassis collision warning method in the present embodiment. The processor executes various functional applications and data processing by running the computer program stored in the memory, that is, the above method is implemented. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the domain controller 20 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0057] The transmission device is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the domain controller 20. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0058] In this embodiment, a vehicle chassis collision warning method is provided. Figure 2 Flowchart of the vehicle chassis collision warning method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0059] Step S210, based on the image acquisition device of the vehicle, acquiring target images in different virtual perspectives in front of the vehicle;

[0060] Step S220, inputting the target images under different virtual perspectives into the deep neural network model for inference to obtain target prediction information of the target obstacle in the original image of the target image;

[0061] Step S230, generating warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle state information; and performing a warning operation based on the warning information.

[0062] Specifically, the image acquisition device includes but is not limited to a camera, a video camera, etc. It is installed on the vehicle and can capture the target image at different virtual perspectives in front of the vehicle; or, capture the original image in front of the vehicle; and then process the original image to obtain the target image at different virtual perspectives. Among them, the different virtual perspectives can be the original perspective (affected by the installation of the image acquisition device, generally the front perspective is used as the original perspective), the top-down perspective, and the columnar head perspective. For example: by setting three cameras with corresponding perspectives, the images corresponding to the original perspective, the top-down perspective, and the columnar head perspective can be obtained respectively. For another example: by setting a camera with an original perspective, the original image of the original perspective can be obtained, and then based on the processing of the original image, the images of the other two perspectives can be obtained. There is no limitation on the specific method of obtaining the target images at different virtual perspectives.

[0063] Among them, the deep neural network model is pre-built and fully trained. The deep neural network model can adopt supervised learning neural network, unsupervised learning neural network and reinforcement learning neural network, etc., and there is no restriction on this. Then the target images under different virtual perspectives are input into the above-mentioned deep neural network model for reasoning, and the relevant feature information in the target images under different virtual perspectives (such as: the position information of the depth map and the height information of the columnar expansion map, etc.) is fully utilized to obtain the target prediction information of the accurate target obstacle in the original image. The target prediction information includes but is not limited to height information, position information, etc.

[0064] Among them, vehicle status information includes but is not limited to steering wheel angle, chassis height and driving speed, etc., which can be provided by various sensors in the vehicle, autonomous driving domain controller or cockpit domain controller. Then, based on the target prediction information and vehicle status information, it can be calculated whether the target obstacle will collide with the vehicle on the current driving path, and the corresponding warning information can be generated; the point where the collision will occur on the current driving path is the target position. Then, based on the warning information, early warning operations can be performed, thereby reducing the occurrence of collisions or scratches and improving driving safety.

[0065] In the relevant technology, the implementation scheme is: using sensors installed on the chassis, generating electrical signals through collision, sending them to the battery management domain controller, and then giving warning information to the driver. However, the disadvantage of this scheme is that it can only detect after a collision occurs and is not predictive. Moreover, with the development of autonomous driving, more and more sensors (cameras, millimeter-wave radars, ultrasonic radars, lidars, etc.) are installed around the vehicle body. However, millimeter-wave radars and lidars detect long-distance targets, and millimeter waves detect close-range targets. These radars have a high false detection rate; the camera image provides very rich information within a reasonable range. In the more popular 360-degree surround view system, an ultra-wide-angle camera (fisheye camera) is included. Due to its ultra-large field of view, it takes on more perception tasks during parking and low-speed driving. However, the fisheye camera collects two-dimensional images. How to use the two-dimensional information in the image to estimate the depth and height of low vertical obstacles on the road in front of the car is a technical difficulty. The present application obtains target images in different virtual perspectives in front of the vehicle through a vehicle-based image acquisition device; the target images in different virtual perspectives are input into a deep neural network model for inference to obtain target prediction information of the target obstacle in the original image of the target image; based on the target prediction information and the obtained vehicle status information, early warning information of the target obstacle at the target position is generated; and early warning operations are performed based on the early warning information, which solves the problem that related technologies can only detect after a collision occurs and lack predictiveness, and the target prediction information of the target obstacle is accurately detected in advance, so that early warning operations can be performed, the occurrence of collisions or scratches can be reduced, and driving safety can be improved.

[0066] The following is a detailed description of each of the above steps:

[0067] In some of the embodiments, Figure 3 As shown, the step S210 of acquiring the target image of the vehicle under different virtual perspectives based on the image acquisition device on the vehicle includes the following steps:

[0068] Step S211, based on the image acquisition device on the vehicle, acquiring the original image in the target image of the vehicle and the internal and external parameters of the image acquisition device;

[0069] Step S212, calculating the original image according to the internal parameters and the external parameters to obtain the depth map and the columnar expansion map in the target image.

[0070] Specifically, the external parameters are used to describe the position and posture of the image acquisition device in the world coordinate system, that is, to determine the conversion relationship between the image acquisition device coordinate system and the world coordinate system. They include but are not limited to translation matrix and rotation matrix. The internal parameters are parameters used to describe the internal imaging characteristics of the image acquisition device, reflecting the inherent properties of the lens and the geometric relationship between the imaging plane and the lens. They include but are not limited to focal length, principal point coordinates, distortion parameters, etc.

[0071] Among them, the original image refers to the image captured by the image acquisition device at a fixed position on the vehicle; then according to the internal and external parameters, the perspective conversion calculation of the original image is performed to obtain the depth map and columnar expansion map in the target image. Among them, the depth map can be an image with depth (position) information such as a top view and a right view; the columnar expansion map can be an expanded image with height such as a left view, a right view, and a main view. Furthermore, in order to simplify the calculation amount of subsequent processing, the resolution of the original image, the depth map, and the columnar expansion map can be unified, and the resolution can be above 512. For example: the resolution is 512×512×3, etc. In other embodiments, it can also be implemented with the help of a well-trained model, a LUT parallel solution, or by calling a third-party component, and there is no limitation to this.

[0072] Through this embodiment, with the help of an image acquisition device, the depth map and the columnar expansion map can be quickly and accurately obtained in combination with the internal and external parameters, thereby reducing the loss of effective data.

[0073] In some embodiments, step S220 of inputting the target images under different virtual perspectives into the deep neural network model for inference to obtain target prediction information of the target obstacle in the original image of the target image includes the following steps:

[0074] Step S221, inputting the original image and the depth image of the target image into the first network model of the deep neural network model for inference to obtain a corresponding feature map; the feature map includes first prediction information of the obstacle;

[0075] Step S222, inputting the feature map and the columnar expansion map of the target image into a second network model of the deep neural network model for inference to obtain second prediction information of the obstacle;

[0076] Step S223: A post-processing module based on the deep neural network model performs a fusion judgment based on the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image.

[0077] Specifically, the target image includes an original image, a depth image, and a columnar expansion image. Figure 4As shown, the deep neural network model includes three parts, namely the first network model (PosNet), the second network model (HeightNet) and the post-processing module (Post Process). It can be understood that these three parts are pre-built and fully trained.

[0078] The first network model is connected to the second network module and the post-processing module respectively, and is used to infer the input original image and depth image to obtain the corresponding feature map. That is to say, the original image and the depth image have feature maps generated after inference respectively; each feature map includes the first prediction information of the obstacle, that is, the feature map that identifies the position of the obstacle with abnormal depth in the original image and the depth map; thus, it is known whether there are obstacles in the original image and where these obstacles are.

[0079] The second network module is connected to the post-processing module and is used to infer the input feature map and column expansion map to obtain the second prediction information of the obstacle. The second prediction information includes the second position information (coordinates), the position confidence ppos, the height h and the height confidence ph.

[0080] Among them, the post-processing module is used to perform fusion judgment based on the first prediction information and the second prediction information, compare the position relationship of obstacles under different viewing angles, related confidence, etc., so as to eliminate interference and obtain accurate target prediction information of target obstacles in the original image.

[0081] Through this embodiment, target prediction information of the target obstacle can be obtained quickly and accurately, and interference from non-target obstacles can be effectively eliminated.

[0082] The following is a detailed description of the deep neural network model:

[0083] In some of the embodiments, the first network model includes a first convolution module and multiple layers of codecs; each layer of codecs includes an encoder and a decoder; the first convolution module serves as the top layer; the output of the encoder of this layer serves as the input of the encoder of the next layer and the decoder of this layer; the output of the decoder of this layer serves as the input of the decoder of the next layer.

[0084] Specifically, Figure 5 As shown, the first network model is a U-shaped structure, including the first convolution module (TopConv) at the top layer and multiple layers of encoders and decoders; each layer of encoders and decoders includes an encoder (Encoder) and a decoder (Decoder); the output of the encoder of this layer serves as the input of the encoder of the next layer and the decoder of this layer; the output of the decoder of this layer serves as the input of the decoder of the next layer.

[0085] Among them, Figure 6As shown in the figure, the encoder structure of each layer is the same, and a multi-head self-attention mechanism is used to collect feature maps of different resolutions. For example, the resolution of the input image is 512×512×6 (including the depth map 512×512×3 and the original image 512×512×3, so the third dimension is 6). After passing through the encoder of the first layer, the resolution is reduced to 128×128×N1, and the output of the encoder of this layer is used as the input of the encoder of the next layer. After passing through the encoder of 4 layers, it reaches the first convolution module of the top layer.

[0086] Among them, Figure 7 As shown in the figure, the first convolution module is mainly used to extract high-level semantic information. The input feature map with reduced resolution is connected through several layers of convolution modules (conv, 3×3 convolution) and average pooling modules (AvgPool) residual connections, and then all the extracted semantic feature maps are combined together through the concatenation module (concat) to obtain a 16×16×N top The feature map sequence is then input into the decoder connected to it.

[0087] Among them, Figure 8 Shown and Fig. 9 As shown in the figure, the decoder of this layer accepts the output of the decoder of the previous layer and the output of the corresponding encoder as input. The decoder of each layer includes a windowed fully connected conditional random field (window FC-CRFs), a reorganization module (rearrange) and an upsampling module (UpamplingBlock). Among them, the fully connected conditional random field divides the feature map into several small areas (called windows), and then calculates the fully connected energy E inside each window. Energy E can be regarded as a contribution to the final position (depth) information. The calculation formula of energy E is as follows:

[0088] E(x) = ∑ i ψ u ( i ) + ∑ ij ψ p (x i , j ) (1);

[0089] In formula 1, x i Represents the predicted value of each point (i) on the feature map; x j Represents the predicted value of each point (j) on the feature map; ψ u Represents a unary potential energy function, which is predicted based on the point value on each feature map.

[0090] ψ p The calculation formula is as follows:

[0091] ψ p= μ(x i , j )f(x i , j )g(I i , j )k(p i , j ) (2);

[0092] In formula 2, μ means if they are not the same feature point, μ=1, if they are the same feature point, μ=0; I i Indicates the color of point i on the feature map; I j Indicates the color of point j on the feature map; p i represents the first position information of point i on the feature map; p j Indicates the first position information of point j on the feature map; uses color and position to guide appropriate results; f is a binary potential function. g and k are parameters.

[0093] In this embodiment, the entire feature map is split into h×ω small areas (windows). The purpose of splitting into multiple windows is to reduce the computational complexity. The total complexity of potential energy in each iteration is estimated as follows:

[0094] Ω(Window FC) = hω*Ω(ψ u ) + hω(N 2 -1)*Ω(ψ p ) (3);

[0095] In formula 3, N is the size of the window, Ω(ψu) and Ω(ψp) are the complexity of calculating the unary potential function and the binary potential function respectively. In addition, although multiple windows are divided, theoretically the calculation of the connection between the feature points of different windows in the entire feature map is cut off, so the window division in the feature map needs to be disrupted; therefore, the work of the rearrange module is introduced.

[0096] like Fig.10 As shown, the left figure is the calculation relationship between the points in each window (red) and other points (blue), and the right figure is a schematic diagram of the reorganization module window.

[0097] ψ u (x i )= μ (I,x i ), (4);

[0098] like Fig.11As shown, unaryNet (unary operation unit) is used to calculate the potential energy of a point on the feature map. In formula 4, θ is the parameter of unarynet, that is, a small neural network is used to predict the potential energy of point i. For binary potential energy calculation ψ p (x i,j ), which mainly calculates the connection between two points and integrates color and position information. Specifically, the transformer mechanism can be used to process the potential energy calculation between two points. The process is: first, a query vector q and a key value vector K are calculated from the feature map of each window. The q and k of all windows form a Q matrix and a K matrix. The dot product of Q and K is calculated to obtain the prediction parameter matrix W. Then, X (including input information such as eigenvalues, colors, and positions) is multiplied by the W matrix. A relative embedded position encoding P module is introduced here to provide the relative position of each point in the window and clearly provide the relative position between points. Therefore, the calculation of binary potential energy is as follows: Finally, the optimized X is obtained through an Optimize net network. ′ as input to the next layer.

[0099] Through this embodiment, the input image is processed to obtain a feature map that identifies the position of an obstacle with depth anomaly in the original image and the depth map, that is, the first prediction information containing the obstacle.

[0100] In some of the embodiments, Fig.12 As shown, the second network model includes a second convolution module, a pooling module and a fully connected layer;

[0101] The second convolution module is connected to the pooling module and is used to extract features from the feature map and the columnar expansion map to obtain a feature sequence;

[0102] The pooling module is connected to the fully connected layer and is used to average pool the feature sequence to obtain the feature vector;

[0103] The fully connected layer is used to transform the feature vector to obtain the second prediction information of the obstacle.

[0104] Specifically, the second network model is mainly used to detect the height of obstacles. The second convolution module can use the convolution network of resnet18 to extract features from the feature map and the columnar expansion map to obtain a feature sequence with reduced resolution; then, through the pooling module (Average pooling), the feature sequence is averaged and pooled to obtain a 512*1*1 dimensional feature vector. Finally, after the fully connected layer (FC(10,5)), the feature vector is transformed to obtain a 10*5 vector; that is, the second prediction information of the five obstacles is obtained; the second prediction information includes the second position information (coordinates), position confidence ppos, height h, and height confidence ph.

[0105] Through this embodiment, accurate detection of the height of obstacles is achieved, and the complexity of detection is reduced.

[0106] In some embodiments, performing fusion judgment according to the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image in step S323 includes the following steps:

[0107] Compare the first prediction information with the second prediction information to determine whether the obstacle is at a local extreme point in the depth map;

[0108] If the obstacle is not at the local extreme point in the depth map, the obstacle will be discarded;

[0109] If the obstacle is at a local extreme point in the depth map, the obstacle is taken as the target obstacle and the corresponding target prediction information is output.

[0110] In this embodiment, the results of the first network model and the second network model are fused. First, verify whether the obstacle in the first prediction information of the first network model is at a local extreme point in the depth map (i.e., at the point where the potential energy is maximum) based on the second prediction information of the second network model; if the obstacle is not at a local extreme point in the depth map, the obstacle is discarded; if the obstacle is at a local extreme point in the depth map, the obstacle is taken as a target obstacle, and the height and position corresponding to the target obstacle are output as target prediction information. Then, the warning information of the target obstacle at the target position can be generated in combination with the vehicle status information; and a warning operation is performed based on the warning information.

[0111] Through this embodiment, the results of the first network model and the second network model can be fused, thereby improving the prediction accuracy of the target prediction information and reducing the occurrence of collisions or scratches.

[0112] In some embodiments, performing the warning operation based on the warning information in step S230 includes the following steps:

[0113] Based on the warning information, the vehicle screen is controlled to display an alarm icon, prompt or warning light;

[0114] Or, based on the warning information, controlling the vehicle's horn to play a prompt message;

[0115] Or, based on the warning information, raise the vehicle's suspension height.

[0116] In this embodiment, various warning operations are provided in combination with various devices on the vehicle (control screen, horn, warning light, vibrator in the seat, etc.) to further reduce the occurrence of collision or scratch. For example: based on the warning information, the vehicle screen is controlled to display an alarm icon, prompt or warning light. Another example: based on the warning information, the vehicle horn is controlled to play prompt information. Another example: based on the warning information, the suspension height of the vehicle is increased, etc.

[0117] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0118] In this embodiment, a vehicle chassis collision warning device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0119] Fig.13 : is a structural block diagram of the vehicle chassis collision warning device of this embodiment, such as Fig.13 As shown, the device includes: an acquisition module 210, a reasoning module 220 and an early warning module 230;

[0120] An acquisition module 210 is used to acquire target images in different virtual perspectives in front of the vehicle based on an image acquisition device of the vehicle;

[0121] The reasoning module 220 is used to input the target images under different virtual perspectives into the deep neural network model for reasoning, and obtain the target prediction information of the target obstacle in the original image of the target image;

[0122] The warning module 230 is used to generate warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle status information; and perform warning operations based on the warning information.

[0123] Through the above-mentioned device, the problem that the related technology can only realize detection after the collision occurs and has no predictive ability is solved. The target prediction information of the target obstacle can be accurately detected in advance, so as to perform early warning operations, reduce the occurrence of collisions or scratches, and improve driving safety.

[0124] In some of the embodiments, the acquisition module 210 is further used to acquire the original image in the target image of the vehicle and the internal and external parameters of the image acquisition device based on the image acquisition device on the vehicle;

[0125] According to the internal and external parameters, the original image is calculated to obtain the depth map and columnar expansion map in the target image.

[0126] In some of the embodiments, the inference module 220 is further used to input the original image and the depth image of the target image into the first network model of the deep neural network model for inference to obtain a corresponding feature map; the feature map includes first prediction information of the obstacle;

[0127] Inputting the feature map and the columnar expansion map of the target image into a second network model of the deep neural network model for inference to obtain second prediction information of the obstacle;

[0128] A post-processing module based on a deep neural network model performs fusion judgment based on the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image.

[0129] In some of the embodiments, the first network model includes a first convolution module and multiple layers of codecs; each layer of codecs includes an encoder and a decoder; the first convolution module serves as the top layer; the output of the encoder of this layer serves as the input of the encoder of the next layer and the decoder of this layer; the output of the decoder of this layer serves as the input of the decoder of the next layer.

[0130] In some of the embodiments, the second network model includes a second convolutional module, a pooling module, and a fully connected layer;

[0131] The second convolution module is connected to the pooling module and is used to extract features from the feature map and the columnar expansion map to obtain a feature sequence;

[0132] The pooling module is connected to the fully connected layer and is used to average pool the feature sequence to obtain the feature vector;

[0133] The fully connected layer is used to transform the feature vector to obtain the second prediction information of the obstacle.

[0134] In some of the embodiments, the reasoning module 220 is further used to compare the first prediction information with the second prediction information to determine whether the obstacle is located at a local extreme point in the depth map;

[0135] If the obstacle is not at the local extreme point in the depth map, the obstacle will be discarded;

[0136] If the obstacle is at a local extreme point in the depth map, the obstacle is taken as the target obstacle and the corresponding target prediction information is output.

[0137] In some of the embodiments, the warning module 230 is further used to control the vehicle screen to display an alarm icon, prompt or warning light based on the warning information;

[0138] Or, based on the warning information, controlling the vehicle's horn to play a prompt message;

[0139] Or, based on the warning information, raise the vehicle's suspension height.

[0140] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0141] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0142] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0143] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0144] S1, based on the image acquisition device of the vehicle, acquiring target images in different virtual perspectives in front of the vehicle;

[0145] S2, input the target images under different virtual perspectives into the deep neural network model for inference, and obtain the target prediction information of the target obstacle in the original image of the target image;

[0146] S3, generating warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle status information; and performing a warning operation based on the warning information.

[0147] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0148] In addition, in combination with the vehicle chassis collision warning method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the vehicle chassis collision warning methods in the above embodiments is implemented.

[0149] It should be noted that the information and data involved in this application are all authorized by the user or fully authorized by all parties and will be used legally.

[0150] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0151] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0152] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0153] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A vehicle chassis collision warning method, characterized in that: include: Based on the image acquisition device of the vehicle, a target image under different virtual viewing angles in front of the vehicle is acquired; Inputting the target images under different virtual perspectives into a deep neural network model for inference to obtain target prediction information of target obstacles in the original image of the target image; Generate warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle state information; And perform warning operations based on the warning information.

2. The vehicle chassis collision warning method according to claim 1, characterized in that: The image acquisition device on the vehicle acquires the target image under different virtual perspectives of the vehicle, including: Based on an image acquisition device on a vehicle, an original image in the target image of the vehicle and an internal parameter and an external parameter of the image acquisition device are acquired; The original image is calculated according to the internal parameters and the external parameters to obtain a depth map and a columnar expansion map in the target image.

3. The vehicle chassis collision warning method according to claim 1, characterized in that: The inputting the target images under different virtual perspectives into a deep neural network model for inference to obtain target prediction information of target obstacles in the original image of the target image includes: Inputting the original image and the depth image of the target image into the first network model of the deep neural network model for inference to obtain a corresponding feature map; the feature map includes first prediction information of the obstacle; Inputting the feature map and the columnar expansion map of the target image into a second network model of the deep neural network model for inference to obtain second prediction information of the obstacle; A post-processing module based on the deep neural network model performs fusion judgment according to the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image.

4. The vehicle chassis collision warning method according to claim 3, characterized in that: The first network model includes a first convolution module and multiple layers of codecs; each layer of the codec includes an encoder and a decoder; the first convolution module serves as the top layer; the output of the encoder of this layer serves as the input of the encoder of the next layer and the decoder of this layer; the output of the decoder of this layer serves as the input of the decoder of the next layer.

5. The vehicle chassis collision warning method according to claim 3, characterized in that: The second network model includes a second convolution module, a pooling module and a fully connected layer; The second convolution module is connected to the pooling module and is used to extract features from the feature map and the columnar expansion map to obtain a feature sequence; The pooling module is connected to the fully connected layer and is used to perform average pooling on the feature sequence to obtain a feature vector; The fully connected layer is used to transform the feature vector to obtain the second prediction information of the obstacle.

6. The vehicle chassis collision warning method according to claim 3, characterized in that: The step of performing fusion judgment based on the first prediction information and the second prediction information to obtain target prediction information of the target obstacle in the original image includes: Compare the first prediction information with the second prediction information to determine whether the obstacle is at a local extreme point in the depth map; If the obstacle is not located at a local extreme point in the depth map, the obstacle is discarded; If the obstacle is located at a local extreme point in the depth map, the obstacle is used as the target obstacle, and corresponding target prediction information is output.

7. The vehicle chassis collision warning method according to claim 1, characterized in that: The performing of the warning operation based on the warning information includes: Based on the warning information, controlling the screen of the vehicle to display an alarm icon, prompt or warning light; or, based on the warning information, controlling the horn of the vehicle to play a prompt message; Or, based on the warning information, raising the suspension height of the vehicle.

8. A vehicle chassis collision warning device, characterized in that: include: Acquisition module, reasoning module and warning module; The acquisition module is used to acquire target images in different virtual perspectives in front of the vehicle based on the image acquisition device of the vehicle; The inference module is used to input the target image under different virtual perspectives into a deep neural network model for inference, so as to obtain target prediction information of the target obstacle in the original image of the target image; The warning module is used to generate warning information of the target obstacle at the target position according to the target prediction information and the acquired vehicle state information; And perform warning operations based on the warning information.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the vehicle chassis collision warning method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the vehicle chassis collision warning method described in any one of claims 1 to 7 are implemented.

Images