Vehicle-mounted visual perception method and device, readable storage medium, and electronic device

Abstract

The embodiments of the present disclosure disclose a vehicle-mounted visual perception method and device, a readable storage medium, and an electronic device, wherein the method comprises: acquiring images at multiple consecutive moments by multiple vehicle-mounted cameras arranged at preset positions on a vehicle to obtain multiple image frame sets; wherein each of the image frame sets includes multiple image frames acquired based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment; performing feature extraction on the image frames included in the multiple image frame sets by a coding branch network in a first network model to obtain multiple first feature maps; performing spatial fusion and temporal fusion on the multiple first feature maps by a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system; and identifying the second feature map based on a network model corresponding to a preset perception task to determine a perception result corresponding to the preset perception task.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a vehicle-mounted visual perception method and device, a readable storage medium, and an electronic device. Background Art

[0002] In the field of autonomous driving, the perception results of multiple on-board cameras in the vehicle's external perception system in their respective coordinate systems cannot be directly used in subsequent prediction and regulation systems. It is necessary to fuse the dynamic and static perception results from different perspectives in a certain way and express them uniformly in the vehicle's coordinate system. In the existing technology, the fusion of different perspectives is usually performed separately through post-processing. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a vehicle-mounted visual perception method and device, a readable storage medium, and an electronic device.

[0004] According to one aspect of an embodiment of the present disclosure, a vehicle-mounted visual perception method is provided, including:

[0005] A plurality of vehicle-mounted cameras arranged at preset positions on the vehicle perform image acquisition at a plurality of consecutive moments to obtain a plurality of image frame sets; wherein each of the image frame sets includes a plurality of image frames acquired by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment;

[0006] Extracting features from the image frames included in the plurality of image frame sets by using the encoding branch network in the first network model to obtain a plurality of first feature maps;

[0007] Performing spatial fusion and temporal fusion on the plurality of first feature maps through a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system;

[0008] The second feature map is identified based on the network model corresponding to the preset perception task to determine the perception result corresponding to the preset perception task.

[0009] According to another aspect of an embodiment of the present disclosure, there is provided a vehicle-mounted visual perception device, comprising:

[0010] An image acquisition module, used to acquire images at multiple consecutive moments by using multiple on-board cameras arranged at preset positions on the vehicle, to obtain multiple sets of image frames; wherein each of the image frame sets includes multiple frames of image frames acquired by the same on-board camera, and each frame of the image frames corresponds to a moment;

[0011] An encoding module, configured to extract features from the image frames included in the plurality of image frame sets obtained by the image acquisition module through an encoding branch network in a first network model, to obtain a plurality of first feature maps;

[0012] A fusion module, configured to perform spatial fusion and temporal fusion on the plurality of first feature maps determined by the encoding module through a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system;

[0013] The perception module is used to identify the second feature map determined by the fusion module based on the network model corresponding to the preset perception task, so as to obtain the perception result corresponding to the preset perception task.

[0014] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the vehicle-mounted visual perception method described in any of the above embodiments.

[0015] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is used to read the executable instructions from the memory and execute the instructions to implement the vehicle-mounted visual perception method described in any of the above embodiments.

[0019] Based on the above embodiments of the present disclosure, a vehicle-mounted visual perception method and device, a readable storage medium, and an electronic device are provided. By realizing spatial fusion and temporal fusion within the first neural network, end-to-end learning of the neural network is achieved. Since there is no need for post-processing fusion, the complexity of spatial fusion and temporal fusion of images during post-processing can be effectively avoided, and the situation where the same target is mistakenly identified as multiple targets in post-processing can be avoided.

[0020] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 It is a structural diagram of a first network model provided by an exemplary embodiment of the present disclosure.

[0023] Figure 2 It is a flowchart of a vehicle-mounted visual perception method provided by an exemplary embodiment of the present disclosure.

[0024] Figure 3 This disclosure Figure 2 A schematic diagram of a flow chart of step 203 in the embodiment shown.

[0025] Figure 4 This disclosure Figure 3 A flowchart of step 2031 in the embodiment shown is shown.

[0026] Figure 5a This disclosure Figure 4 A schematic diagram of a flow chart of step 401 in the embodiment shown.

[0027] Figure 5b It is a schematic diagram of the conversion relationship between the bird's-eye view imaging plane and the preset plane of the vehicle coordinate system in an exemplary embodiment of the present disclosure.

[0028] Figure 6 This disclosure Figure 3 A flowchart of step 2032 in the embodiment shown is shown.

[0029] Figure 7 This disclosure Figure 2 A flow chart of step 204 in the embodiment shown is shown.

[0030] Figure 8 It is a schematic diagram of the structure of a vehicle-mounted visual perception device provided by an exemplary embodiment of the present disclosure.

[0031] Fig. 9 It is a schematic diagram of the structure of a vehicle-mounted visual perception device provided by another exemplary embodiment of the present disclosure.

[0032] Fig.10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0034] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0038] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0039] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0040] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0041] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0042] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0045] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0046] Application Overview

[0047] In the process of implementing the present disclosure, the inventors found that the vehicle-mounted visual perception method provided in the prior art usually adopts a post-processing method. After the neural network model outputs the perception results, the perception results of different cameras are fused through some rules; this method has at least the following problems: when the perception results of the overlapping areas of adjacent perspectives are fused through post-processing, due to the overlap between multiple perspectives, detection results will be obtained for the same target in images of different perspectives, and the same target appears in different positions in different images, which requires complex identification and detection during post-processing fusion, and it is easy to identify the same target as multiple targets, causing ambiguity.

[0048] Exemplary network structure

[0049] Figure 1 is a schematic diagram of the structure of a first network model provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the first network model in this embodiment includes: an encoding branch network 101, a fusion branch network 102 and a decoding branch network 103.

[0050] In this embodiment, the image frames captured by multiple vehicle-mounted cameras are feature extracted by the encoding branch network 101 to obtain multiple first feature maps; wherein, the multiple vehicle-mounted cameras are vehicle-mounted cameras set at different preset positions on the vehicle, each vehicle-mounted camera corresponds to a viewing angle, and the shooting field of view corresponding to the multiple vehicle-mounted cameras can cover the surrounding environment of the vehicle. In order to avoid the blind spot of the viewing angle, the shooting field of view between the multiple vehicle-mounted cameras may overlap. During the driving process of the vehicle, each vehicle-mounted camera continuously captures multiple images at multiple times based on a viewing angle at a set position to obtain a set of image frames corresponding to each vehicle-mounted camera. It can be understood that each vehicle-mounted camera captures an image frame at each moment, and each image frame set includes multiple image frames captured by the same vehicle-mounted camera and arranged based on time sequence. Optionally, the encoding branch network 101 can be any network that can realize feature extraction, such as a convolutional neural network, etc. This embodiment does not limit the specific network structure adopted.

[0051] The fusion branch network 102 performs spatial fusion and temporal fusion on the multiple first feature maps to obtain a second feature map in the bird's-eye view image coordinate system; wherein the spatial fusion fuses multiple image frames collected by each of the multiple vehicle-mounted cameras at the same time to obtain a third feature map in the bird's-eye view image coordinate system; the temporal fusion fuses the third feature map corresponding to each moment in the multiple moments to obtain a second feature map after temporal fusion.

[0052] The second feature map obtained by the fusion branch network 102 is decoded by the decoding branch network 103 to obtain a decoded second feature map; optionally, the decoded second feature map in this embodiment can be a feature map that conforms to the structure of subsequent perception tasks. Optionally, the decoding branch network 103 can be any network that can realize feature extraction, such as a convolutional neural network, etc. This embodiment does not limit the specific network structure.

[0053] The first network model provided in this embodiment is for realizing visual perception tasks. Therefore, any visual perception task can be connected after the first network model. The visual perception task may include but is not limited to: segmentation tasks, detection tasks, classification tasks, etc.

[0054] This embodiment realizes spatial fusion and temporal fusion within the first network model, so that the disturbance of the internal and external parameters of the camera caused by the change of the camera posture during the vehicle driving process does not directly affect the perception result. The first network model is trained and learned to overcome the influence of the disturbance, so that the output perception result is smoother and not affected by the change of the camera posture during the vehicle driving process; spatial fusion and temporal fusion are realized within the first network model, and there is no need to perform spatial fusion and temporal fusion in the post-processing process, thus avoiding fusion in post-processing and reducing the complexity of post-processing, so that the first network model can realize end-to-end learning and has the potential for joint learning with candidate visual perception tasks. In addition, the perception fusion method through the first network model provided in the present application has lower requirements on the computational complexity of the chip.

[0055] Exemplary Methods

[0056] Figure 2 FIG. 1 is a flow chart of a vehicle-mounted visual perception method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 2 As shown, the following steps are included:

[0057] Step 201 , multiple vehicle-mounted cameras arranged at preset positions on a vehicle perform image acquisition at multiple consecutive moments to obtain multiple image frame sets.

[0058] Each image frame set includes multiple image frames collected by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment.

[0059] Optionally, since each vehicle-mounted camera corresponds to an image frame set, multiple vehicle-mounted cameras obtain multiple image frame sets; each vehicle-mounted camera can correspond to different directions, and the image acquisition ranges corresponding to multiple vehicle-mounted cameras may or may not overlap. Normally, by setting multiple vehicle-mounted cameras at multiple preset positions, the vehicle's corresponding all-round angles can be photographed to provide more information for autonomous driving or assisted driving.

[0060] Step 202: extract features from image frames included in a plurality of image frame sets through an encoding branch network in a first network model to obtain a plurality of first feature maps.

[0061] Optionally, the encoding branch network in the first network model sequentially extracts features from the image frames included in the plurality of image frame sets to obtain a plurality of first feature maps; the encoding branch network in this embodiment can refer to Figure 1 The provided encoding branch network 101 is used to understand and realize feature extraction of the image frame.

[0062] Step 203: Perform spatial fusion and temporal fusion on the multiple first feature maps through the fusion branch network in the first network model to obtain a second feature map in the bird's-eye view image coordinate system.

[0063] Optionally, the coding branch network in this embodiment can refer to Figure 1 The provided fusion branch network 102 is used to understand and realize the fusion of the first feature map.

[0064] Step 204: Identify the second feature map based on the network model corresponding to the preset perception task to determine the perception result corresponding to the preset perception task.

[0065] The preset perception task in this embodiment can be any visual perception task, for example, a segmentation task, a detection task, a classification task, etc., and the recognition operation of this step is implemented by a network model corresponding to the visual perception task.

[0066] The above-mentioned embodiment of the present disclosure provides an in-vehicle visual perception method, which realizes end-to-end learning of the neural network by realizing spatial fusion and temporal fusion within the first neural network. Since there is no need to adopt post-processing fusion, the complexity of spatial fusion and temporal fusion of images in post-processing can be effectively avoided, and the situation where the same target is mistakenly identified as multiple targets in post-processing can be avoided.

[0067] like Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step 203 may include the following steps:

[0068] Step 2031: For each of the multiple moments, perform spatial fusion on the multiple first feature maps corresponding to the moment to obtain multiple third feature maps in the bird's-eye view image coordinate system.

[0069] Among them, each third characteristic map corresponds to a moment.

[0070] In this embodiment, the image frames captured by each vehicle-mounted camera are image frames in the camera coordinate system corresponding to the vehicle-mounted camera. In order to project multiple first feature maps to the bird's-eye image coordinate system respectively before fusion, and then fuse the multiple feature maps in the bird's-eye image coordinate system, the fusion of multiple feature maps in the same coordinate system can be achieved through the feature map fusion method in the relevant technology, for example, element-by-element addition, splicing in the feature channel dimension, taking the maximum value of each element, neural network fusion, etc.

[0071] Step 2032: Perform temporal fusion on multiple third feature maps to obtain a second feature map.

[0072] In this embodiment, each third feature map corresponds to a moment. By reconstructing the third feature map corresponding to each moment into the third feature map corresponding to a certain moment (for example, the most recent moment corresponding to the last image frame collected), the second feature map can be obtained by fusing the reconstructed multiple third feature maps. The fusion in this step can also be achieved by the feature map fusion method in the related technology, for example, element-by-element addition, splicing in the feature channel dimension, taking the maximum value element-by-element, neural network fusion, etc.; in this embodiment, spatial fusion is performed first, and then temporal fusion is performed, so that temporal fusion is easier to implement and the speed of feature map fusion is accelerated.

[0073] like Figure 4 As shown in the above Figure 3 Based on the illustrated embodiment, step 2031 may include the following steps:

[0074] Step 401 , performing homography transformation on each of the multiple first feature maps respectively to obtain multiple transformed feature maps in a bird's-eye view image coordinate system.

[0075] Among them, homography transformation (or projection transformation) has 8 degrees of freedom and is used to describe the mapping relationship between points on two planes. By transforming each first feature map in the image coordinate system to the bird's-eye view image coordinate system through homography transformation, multiple transformed feature maps can be obtained.

[0076] Step 402: fuse multiple transformed feature maps point by point to obtain a third feature map in the bird's-eye view image coordinate system.

[0077] In this embodiment, multiple transformation feature maps in the same coordinate system are obtained through homography transformation, and the sizes of the multiple transformation feature maps are the same. Therefore, the fusion of the multiple transformation feature maps can be achieved through point-by-point fusion. The point-by-point fusion method may include but is not limited to: element-by-element addition, splicing in the feature channel dimension, element-by-element maximum value, neural network fusion, etc.; in this embodiment, multiple first feature maps corresponding to different coordinate systems are mapped to the same coordinate system through homography transformation, the feature map of each perspective is converted from the camera perspective to the bird's-eye view perspective, and fusion is achieved in the same coordinate system, so that the fused third feature map can have image features corresponding to all angles around the vehicle. When performing visual perception tasks based on the third feature map, the perception results of different vehicle-mounted cameras can be fused without post-processing, which improves the fusion efficiency of the perception results and reduces the difficulty of fusion.

[0078] like Figure 5a As shown in the above Figure 4 Based on the illustrated embodiment, step 401 may include the following steps:

[0079] Step 4011, based on the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle camera corresponding to each first feature map, determine the first transformation matrix from the vehicle coordinate system to the camera coordinate system corresponding to the first feature map.

[0080] In this embodiment, the intrinsic parameter matrix corresponding to each vehicle-mounted camera is a known 3*3 matrix K. The extrinsic parameter matrix corresponding to the vehicle-mounted camera can be calculated by setting the vehicle-mounted camera at a preset position on the vehicle. Based on the intrinsic parameter matrix and the extrinsic parameter matrix, the first transformation matrix T between the vehicle coordinate system and the camera coordinate system can be determined. vcs2cam , the first transformation matrix includes rotation parameters and translation parameters. For example, in an optional example, the first transformation matrix can be expressed as: Among them, r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 、r 33 represents the rotation parameter, and t1, t2, and t3 represent the translation parameters.

[0081] Step 4012: determine a second transformation matrix from the bird's-eye view image coordinate system to the preset plane of the ego-vehicle coordinate system based on the perception range on the preset plane of the ego-vehicle coordinate system and the scaling ratio and translation distance of the bird's-eye view image coordinate system.

[0082] Alternatively, if Figure 5b As shown in the figure, d1, d2, d3, and d4 represent the coordinate system of the vehicle on the preset plane (xy plane, with O in the figure) vcs represents the center point of the xy plane in the ego-vehicle coordinate system), based on which the translation distance of the ego-vehicle coordinate system relative to the bird's-eye view image coordinate system can be determined (e.g. Figure 5b d1, d3), assuming that the bird's-eye view imaging plane (the center point of the imaging plane can be Figure 5b O BEV ) coincides with (or is parallel to) the xy plane of the ego-vehicle coordinate system, and r represents the scaling ratio from the xy plane of the ego-vehicle coordinate system to the bird's-eye view imaging plane. The second transformation matrix can be expressed as:

[0083] Step 4013, based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera, determine a third transformation matrix from the bird's-eye view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera.

[0084] Optionally, in this embodiment, the transformation matrix from the bird's-eye view image coordinate system to the camera coordinate system can be derived through the first transformation matrix and the second transformation matrix, and then combined with the intrinsic parameter matrix, the third transformation matrix T between the bird's-eye view image coordinate system and the image plane (corresponding image coordinate system) corresponding to the vehicle camera can be determined. bev2img .

[0085] Step 4014: transform the first feature map based on the third transformation matrix to obtain a transformed feature map.

[0086] In this embodiment, the third transformation matrix is ​​determined by the above determination, that is, the conversion relationship between the image coordinate system and the bird's-eye view image coordinate system is determined. Through this conversion relationship, the feature of the image position (u1, v1) can be reversely indexed according to the bird's-eye view image (u, v) position. For example, feature mapping can be implemented by the following formula (1):

[0087]

[0088] Thus, the transformed feature map f after the perspective conversion is obtained n Similarly, the above transformation operation is performed on the first feature map corresponding to each viewing angle to obtain N (N is the number of vehicle-mounted cameras, which can be set according to the specific scene) transformed feature maps {f n}, n = 1, 2, ..., N; finally, perform feature fusion to obtain the third feature map F at time t (one of the multiple time points) after spatial fusion t .

[0089] Optionally, based on the above embodiment, step 3013 may further include:

[0090] a1, based on the first transformation matrix, determine the fourth transformation matrix from the preset plane of the vehicle coordinate system to the camera coordinate system.

[0091] Optionally, assuming that the imaging plane of the bird's-eye view coincides with (or is parallel to) the xy plane of the ego-vehicle coordinate system, the xy plane being the plane where the z value of the three-dimensional point coordinate in the ego-vehicle coordinate system is equal to 0 (or equal to any same value), when they coincide, the first transformation matrix T can be simply eliminated. vcs2cam The column corresponding to the z axis (column 3) in , when parallel, the third column can be set to the preset value to obtain the fourth transformation matrix T between the imaging plane of the bird's-eye view image coordinate system and the camera coordinate system vcs_xy2cam For example, in an optional example (the imaging plane of the bird's-eye view image coordinate system coincides with the xy plane of the vehicle coordinate system), the fourth transformation matrix can be expressed as:

[0092] a2. Determine a fifth transformation matrix from the bird's-eye view image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix.

[0093] Optionally, after determining the fourth transformation matrix, the fifth transformation matrix between the bird's-eye view image coordinate system and the camera coordinate system can be determined by performing matrix multiplication on the fourth transformation matrix and the second transformation matrix in combination with the second transformation matrix. For example, the fifth transformation matrix T can be determined based on the following formula (2): bev2cam :

[0094] T bev2cam =T vcs-xy2cam ×T bev2vcs-xy Formula (2)

[0095] a3. Determine a third transformation matrix based on the fifth transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera.

[0096] In this embodiment, it is determined that the number of rows and columns between the fifth transformation matrix and the intrinsic parameter matrix do not correspond. In order to perform matrix multiplication (the intrinsic parameter matrix is ​​a 3*3 matrix), a truncation operation may be performed on the fifth transformation matrix, and the first three rows of the fifth transformation matrix may be truncated to obtain a truncated transformation matrix. For example, the fifth transformation matrix T bev2cam The first three rows of are intercepted to obtain the interception transformation matrix T′ bev2cam At this time, the third transformation matrix is ​​determined by performing matrix multiplication on the intercepted transformation matrix and the intrinsic parameter matrix, for example, as shown in the following formula (3):

[0097] T bev2img =K×T′ bev2cam Formula (3)

[0098] The third transformation matrix can be obtained through the above processing. Based on the third transformation matrix, the first feature map of the corresponding image coordinate system can be converted to the imaging plane in the bird's-eye image coordinate system. In this embodiment, the third transformation matrix for converting the bird's-eye image coordinate system to the image coordinate system is determined through the coordinate system relationship conversion. It is realized that based on the third transformation matrix, the features in the image coordinate system can be directly converted to the bird's-eye image coordinate system, and fast spatial fusion is realized in the fusion branch model of the first network model. The specific parameters in the coordinate system conversion are determined by learning the first network model, which improves the accuracy and speed of the conversion.

[0099] like Figure 6 As shown in the above Figure 3 Based on the illustrated embodiment, step 2032 may include the following steps:

[0100] Step 601: Use the third feature map corresponding to the latest moment among multiple moments as a reference feature map.

[0101] Optionally, when multiple moments include moments t, t-1...ts, the third feature map corresponding to the tth moment (the most recent moment) can be used as a reference feature map. Of course, this embodiment is only a specific example. In actual applications, the third feature map corresponding to any moment can be used as a reference feature map, which does not affect the operation and results of the time series fusion. When the most recent moment is selected as the parameter feature map, the result of the time domain fusion corresponds to the most recent moment, that is, the output second feature corresponds to the most recent moment (for example, the current moment), which improves the real-time performance of the perception result determined based on the second feature, that is, the real-time performance of the vehicle-mounted perspective perception method is improved.

[0102] Step 602: reconstruct each third feature map in the at least one third feature map to obtain at least one fourth feature map.

[0103] In this embodiment, in order to realize the fusion of the third feature maps corresponding to multiple moments, each third feature map is mapped to the feature space corresponding to the reference feature map, so as to perform feature fusion.

[0104] Step 603: perform point-by-point fusion on the reference feature map and at least one fourth feature map to obtain a second feature map.

[0105] Optionally, point-by-point fusion methods may include but are not limited to: element-by-element addition, splicing in the feature channel dimension, element-by-element maximum value, neural network fusion, etc.; this embodiment realizes spatial fusion and temporal fusion in the fusion branch network of the first network model in sequence by temporal fusion of the third feature map corresponding to each of the multiple moments obtained after spatial fusion, thereby achieving internal branch realization in the first network model, fusing multiple image frames collected from multiple perspectives at multiple moments into a second feature map in the bird's-eye view image coordinate system, so that the first network model can learn end-to-end how to achieve spatial and temporal fusion of feature maps, without the need for external post-processing after the network model to perform spatial and temporal fusion, thereby reducing the complexity of post-processing and reducing the ambiguity problems introduced by fusion in post-processing.

[0106] Optionally, based on the above embodiment, step 602 may further include:

[0107] b1, based on the inter-frame motion transformation matrix corresponding to at least one moment of the vehicle-mounted camera at the most recent moment and at least one third feature map, determine at least one homography transformation matrix between the first feature map corresponding to the reference feature map and the first feature map corresponding to at least one third feature map.

[0108] Optionally, since the position of the vehicle may change when the on-board camera collects images at different times, resulting in a corresponding change in the camera position, the homography transformation matrix H between the two frames before and after any viewpoint is calculated based on the vehicle's motion between frames (one image frame corresponds to each moment) and the height d of the on-board camera (e.g., the on-board front-view camera) at the set position of the vehicle from the xy plane of the vehicle coordinate system is known (it is a camera extrinsic parameter and can be calculated and determined by an existing algorithm or by information collected by the sensor, etc.). Assume that the rotation from time t to time t-1 is R and the translation is m. At this time, the homography transformation matrix H can be determined based on the following formula (4):

[0109]

[0110] Where n = [0, 0, 1] T is the normal of the xy plane of the ego-vehicle coordinate system; K represents the intrinsic parameter matrix of the vehicle camera; d represents the height of the vehicle camera from the xy plane of the ego-vehicle coordinate system.

[0111] b2. Determine at least one conversion matrix between a reference feature map and at least one third feature map based on at least one homography transformation matrix and a third transformation matrix.

[0112] In this embodiment, a transformation matrix is ​​determined for each third feature map by combining the third transformation matrix of the spatial transformation and the homography transformation matrix corresponding to each third feature map. Optionally, the transformation matrix T can be determined based on the following formula (5): temp :

[0113]

[0114] b3, reconstructing each third feature map in the at least one third feature map based on each conversion matrix in the at least one conversion matrix to obtain at least one fourth feature map.

[0115] In this embodiment, taking the third feature map at time t-1 as an example, after spatial fusion at time t-1, (U t-1 , V t-1 ) position features can be obtained by the transformation matrix T temp Mapped to the spatial fusion feature at time t (U t , V t ), thus obtaining the reconstructed feature F′ at time t tThat is, the third feature map at time t-1 is reconstructed through the conversion matrix to obtain the fourth feature map, and so on, the third feature map at each time is reconstructed to obtain at least one fourth feature map; through feature map reconstruction, the feature maps corresponding to different times can be fused, and, by performing spatial fusion and then performing temporal fusion, the number of feature maps processed by temporal fusion is reduced, the difficulty of temporal fusion is reduced, and, by combining the third transformation matrix in spatial fusion to realize temporal fusion, the reuse rate of parameters is improved, and the fusion efficiency is improved.

[0116] like Figure 7 As shown in the above Figure 2 Based on the illustrated embodiment, step 204 may include the following steps:

[0117] Step 2041: decode the second feature map through the decoding branch network in the first network model to obtain a decoded second feature map.

[0118] Optionally, the decoding branch network in this embodiment can refer to Figure 1 The decoding branch network 103 in the embodiment is provided for understanding.

[0119] Step 2042: Identify the decoded second feature map based on the second network model to determine the perception result corresponding to the preset perception task.

[0120] In this embodiment, the second feature map can be mapped to the feature space required by the preset perception task through the decoding branch network, so that the network model corresponding to the subsequent preset perception task can directly process the mapped feature map, reducing other intermediate processing processes, and realizing that the network model of the preset perception task is directly connected to the first network model, and the joint learning of the network models can be realized. Through joint learning, the accuracy of the perception results of the preset perception task is improved.

[0121] Any vehicle-mounted visual perception method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any vehicle-mounted visual perception method provided in the embodiments of the present disclosure may be executed by a processor, such as a processor executing any vehicle-mounted visual perception method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0122] Exemplary Devices

[0123] Figure 8 FIG. 1 is a schematic diagram of the structure of a vehicle-mounted visual perception device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the device provided in this embodiment includes:

[0124] The image acquisition module 81 is used to acquire images at multiple consecutive moments through multiple vehicle-mounted cameras arranged at preset positions on the vehicle to obtain multiple image frame sets.

[0125] Each image frame set includes multiple image frames collected by the same vehicle-mounted camera, and each of the vehicle-mounted cameras corresponds to one image frame at each moment.

[0126] The encoding module 82 is used to extract features from the image frames included in the multiple image frame sets obtained by the image acquisition module 81 through the encoding branch network in the first network model to obtain multiple first feature maps.

[0127] The fusion module 83 is used to perform spatial fusion and temporal fusion on the multiple first feature maps determined by the encoding module 82 through the fusion branch network in the first network model to obtain a second feature map in the bird's-eye view image coordinate system.

[0128] The perception module 84 is used to identify the second feature map determined by the fusion module 83 based on the network model corresponding to the preset perception task, and obtain the perception result corresponding to the preset perception task.

[0129] The above-mentioned embodiment of the present disclosure provides a vehicle-mounted visual perception device, which realizes end-to-end learning of the neural network by realizing spatial fusion and temporal fusion within the first neural network. Since there is no need to adopt post-processing fusion, the complexity of spatial fusion and temporal fusion of images in post-processing can be effectively avoided, and the situation of misidentifying the same target as multiple targets in post-processing can be avoided.

[0130] Fig. 9 FIG. 1 is a schematic diagram of the structure of a vehicle-mounted visual perception device provided by another exemplary embodiment of the present disclosure. Fig. 9 As shown, in the device provided in this embodiment, the fusion module 83 includes:

[0131] The spatial fusion unit 831 is used to perform spatial fusion on the multiple first feature maps corresponding to each moment in the multiple moments, so as to obtain multiple third feature maps in the bird's-eye view image coordinate system.

[0132] Among them, each third characteristic map corresponds to a moment.

[0133] The time series fusion unit 832 is used to perform time series fusion on multiple third feature maps to obtain a second feature map.

[0134] Optionally, the spatial fusion unit 831 is specifically used to perform homography transformation on each of the multiple first feature maps respectively to obtain multiple transformed feature maps in the bird's-eye view image coordinate system; and perform point-by-point fusion on the multiple transformed feature maps to obtain a third feature map in the bird's-eye view image coordinate system.

[0135] Optionally, when the spatial fusion unit 831 performs homography transformation on each of the multiple first feature maps respectively to obtain multiple transformed feature maps in the bird's-eye image coordinate system, it is used to determine the first transformation matrix from the vehicle coordinate system to the camera coordinate system corresponding to the first feature map based on the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle-mounted camera corresponding to each first feature map; determine the second transformation matrix from the bird's-eye image coordinate system to the preset plane of the vehicle coordinate system based on the perception range on the preset plane of the vehicle coordinate system and the scaling ratio and translation distance of the bird's-eye image coordinate system; determine the third transformation matrix from the bird's-eye image coordinate system to the image coordinate system corresponding to the vehicle camera based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera; transform the first feature map based on the third transformation matrix to obtain a transformed feature map.

[0136] Optionally, when the spatial fusion unit 831 determines the third transformation matrix from the bird's-eye view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera, it is used to determine the fourth transformation matrix from the preset plane of the vehicle coordinate system to the camera coordinate system based on the first transformation matrix; determine the fifth transformation matrix from the bird's-eye view image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix; and determine the third transformation matrix based on the fifth transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera.

[0137] Optionally, the timing fusion unit 832 is specifically used to use the third feature map corresponding to the most recent moment among multiple moments as a reference feature map; reconstruct each third feature map in at least one third feature map respectively to obtain at least one fourth feature map; and perform point-by-point fusion of the reference feature map and at least one fourth feature map to obtain a second feature map.

[0138] Optionally, when the timing fusion unit 832 reconstructs each third feature map in at least one third feature map respectively to obtain at least one fourth feature map, it is used to determine at least one homography transformation matrix between the first feature map corresponding to the reference feature map and the first feature map corresponding to at least one third feature map based on the inter-frame motion transformation matrix corresponding to the most recent moment and at least one moment corresponding to the at least one third feature map of the vehicle-mounted camera; determine at least one transformation matrix between the reference feature map and at least one third feature map based on at least one homography transformation matrix and the third transformation matrix; and reconstruct each third feature map in at least one third feature map based on each transformation matrix in the at least one transformation matrix to obtain at least one fourth feature map.

[0139] In some optional embodiments, the perception module 84 includes:

[0140] The decoding unit 841 is used to decode the second feature map through the decoding branch network in the first network model to obtain a decoded second feature map.

[0141] The feature recognition unit 842 is used to recognize the decoded second feature map based on the second network model to determine the perception result corresponding to the preset perception task.

[0142] Exemplary Electronic Devices

[0143] Below, reference Fig.10 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device 100 and the second device 200, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0144] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0145] like Fig.10 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0146] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0147] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the vehicle-mounted visual perception method of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0148] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0149] For example, when the electronic device is the first device 100 or the second device 200, the input device 13 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, for receiving collected input signals from the first device 100 and the second device 200.

[0150] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0151] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0152] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0153] Exemplary computer program products and computer-readable storage media

[0154] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the in-vehicle visual perception method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0155] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0156] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the in-vehicle visual perception method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0157] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0158] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0159] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0161] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0162] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0163] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0164] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A vehicle-mounted visual perception method, comprising: A plurality of vehicle-mounted cameras arranged at preset positions on the vehicle perform image acquisition at a plurality of consecutive moments to obtain a plurality of image frame sets; wherein each of the image frame sets includes a plurality of image frames acquired by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment; Extracting features from the image frames included in the plurality of image frame sets by using the encoding branch network in the first network model to obtain a plurality of first feature maps; Performing spatial fusion and temporal fusion on the plurality of first feature maps through a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system; Identify the second feature map based on a network model corresponding to a preset perception task, and determine a perception result corresponding to the preset perception task; The performing spatial fusion and temporal fusion on the plurality of first feature maps through the fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system includes: For each of the multiple moments, perform spatial fusion on the multiple first feature maps corresponding to the moment to obtain multiple third feature maps in the bird's-eye view image coordinate system; wherein each of the third feature maps corresponds to a moment; Using a third characteristic graph corresponding to a most recent moment among the multiple moments as a reference characteristic graph; Reconstruct each of the at least one third feature map respectively to obtain at least one fourth feature map; The reference feature map and the at least one fourth feature map are point-by-point merged to obtain the second feature map.

2. The method according to claim 1, wherein: The performing spatial fusion on the multiple first feature maps corresponding to the moment to obtain multiple third feature maps in the bird's-eye view image coordinate system includes: Performing homography transformation on each of the multiple first feature maps respectively to obtain multiple transformed feature maps in a bird's-eye view image coordinate system; The multiple transformed feature maps are fused point by point to obtain the third feature map in the bird's-eye view image coordinate system.

3. The method according to claim 2, wherein: The performing homography transformation on each of the plurality of first feature maps to obtain a plurality of transformed feature maps in the bird's-eye view image coordinate system comprises: Determine a first transformation matrix from a vehicle coordinate system to a camera coordinate system corresponding to each of the first feature maps based on an intrinsic parameter matrix and an extrinsic parameter matrix of the vehicle camera corresponding to each of the first feature maps; Determine a second transformation matrix from the bird's-eye view image coordinate system to the preset plane of the ego-vehicle coordinate system based on the perception range on the preset plane of the ego-vehicle coordinate system and the scaling ratio and translation distance of the bird's-eye view image coordinate system; Determine a third transformation matrix from the bird's-eye view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera; The first feature map is transformed based on the third transformation matrix to obtain the transformed feature map.

4. The method according to claim 3, wherein: The determining, based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the vehicle-mounted camera, a third transformation matrix from the bird's-eye view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera comprises: Based on the first transformation matrix, determining a fourth transformation matrix from the preset plane of the vehicle coordinate system to the camera coordinate system; Determine a fifth transformation matrix from the bird's-eye view image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix; The third transformation matrix is ​​determined based on the fifth transformation matrix and the intrinsic parameter matrix of the vehicle-mounted camera.

5. The method according to claim 4, wherein: The reconstructing each of the at least one third feature map to obtain at least one fourth feature map comprises: Determine at least one homography transformation matrix between a first feature map corresponding to the reference feature map and a first feature map corresponding to the at least one third feature map based on an inter-frame motion transformation matrix corresponding to at least one moment of the vehicle-mounted camera at a recent moment and corresponding to the at least one third feature map; determining, based on the at least one homography transformation matrix and the third transformation matrix, at least one transformation matrix between the reference feature map and the at least one third feature map; Each of the third feature maps in the at least one third feature map is reconstructed based on each of the conversion matrices in the at least one conversion matrix to obtain the at least one fourth feature map.

6. The method according to any one of claims 1 to 5, wherein: The identifying the second feature graph based on the second network model corresponding to the preset perception task to determine the perception result corresponding to the preset perception task includes: Decoding the second feature map through a decoding branch network in the first network model to obtain a decoded second feature map; The decoded second feature map is identified based on the second network model to determine the perception result corresponding to the preset perception task.

7. A vehicle-mounted visual perception device, comprising: An image acquisition module, used to acquire images at multiple consecutive moments by using multiple on-board cameras arranged at preset positions on the vehicle, to obtain multiple sets of image frames; wherein each of the image frame sets includes multiple image frames acquired by the same on-board camera, and each of the on-board cameras corresponds to one image frame at each moment; An encoding module, configured to extract features from the image frames included in the plurality of image frame sets obtained by the image acquisition module through an encoding branch network in a first network model, to obtain a plurality of first feature maps; A fusion module, configured to perform spatial fusion and temporal fusion on the plurality of first feature maps determined by the encoding module through a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system; A perception module, used to identify the second feature map determined by the fusion module based on a network model corresponding to a preset perception task, and obtain a perception result corresponding to the preset perception task; The fusion module includes: a spatial fusion unit, configured to perform spatial fusion on a plurality of first feature maps corresponding to each of the plurality of moments, to obtain a plurality of third feature maps in a bird's-eye view image coordinate system; wherein each of the third feature maps corresponds to a moment; A time series fusion unit is used to use the third feature map corresponding to the most recent moment among the multiple moments as a reference feature map; reconstruct each of the at least one third feature map respectively to obtain at least one fourth feature map; and perform point-by-point fusion on the reference feature map and the at least one fourth feature map to obtain the second feature map.

8. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the vehicle-mounted visual perception method described in any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the vehicle-mounted visual perception method described in any one of claims 1-6 above.

Images