Visual perception method and device, readable storage medium and electronic equipment

Abstract

A visual perception method is disclosed, comprising: acquiring a plurality of images respectively corresponding to a plurality of perspectives collected by a multi-camera system of a carrier at the same position; encoding the plurality of images respectively to obtain image features respectively corresponding to the images; decomposing the image features in a remapping manner to obtain overhead perspective features and surround perspective features respectively corresponding to the images; fusing the overhead perspective features and the surround perspective features to obtain three-dimensional space features; decoding the three-dimensional space features to obtain a three-dimensional space structure; and realizing visual perception according to the three-dimensional space structure. The visual perception method, device, readable storage medium and electronic equipment provided by the present application can decompose images into surround view features and overhead perspective features, and can obtain accurate three-dimensional space features after fusion, thereby improving the perception effect.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a visual perception method, device, readable storage medium, and electronic device. Background Art

[0002] 360-degree perception based on surround-view images is a key technology for autonomous driving. The key technology to be addressed is how to fully utilize the information in surround-view images for autonomous driving perception. Existing technologies primarily fuse surround-view images through inverse perspective transformation, using methods such as front fusion, mid-fusion, and back fusion. However, existing inverse perspective transformation-based stitching schemes assume a flat ground surface, which provides good perception of objects near the road. However, for objects at a certain height, such as traffic lights and signs, severe distortion occurs, resulting in a loss of vertical spatial information, which affects perception. Summary of the Invention

[0003] This application is proposed to address the above technical issues. The embodiments of this application provide a visual perception method, apparatus, readable storage medium, and electronic device that decompose an image into surround view features and top-down perspective features. After fusion, these features can obtain accurate three-dimensional spatial features and improve perception.

[0004] According to one aspect of the present application, a visual perception method is provided, comprising:

[0005] Acquire multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position;

[0006] Encoding the multiple images respectively to obtain image features corresponding to the images;

[0007] Decomposing each of the image features by remapping to obtain a top-view perspective feature and a surround-view perspective feature corresponding to each of the images;

[0008] Fusing the top-view features and the surround-view features to obtain three-dimensional spatial features;

[0009] Decoding the three-dimensional spatial features to obtain a three-dimensional spatial structure;

[0010] Visual perception is achieved based on the three-dimensional spatial structure.

[0011] According to a second aspect of the present application, a visual perception device is provided, comprising:

[0012] An acquisition module, configured to acquire multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position;

[0013] An encoding module, configured to encode the plurality of images respectively to obtain image features corresponding to the images;

[0014] a transformation module, configured to decompose each of the image features by remapping to obtain a top-view feature and a surround-view feature corresponding to each of the images;

[0015] A fusion module, configured to fuse the top-view features and the surround-view features to obtain a three-dimensional spatial feature;

[0016] A decoding module, configured to decode the three-dimensional spatial features to obtain a three-dimensional spatial structure;

[0017] The perception module is used to realize visual perception based on the three-dimensional space structure.

[0018] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute any of the above-mentioned visual perception methods.

[0019] According to a fourth aspect of the present application, an electronic device is provided, comprising:

[0020] processor;

[0021] a memory for storing instructions executable by the processor;

[0022] The processor is used to read the executable instructions from the memory and execute the instructions to implement any of the above-mentioned visual perception methods.

[0023] In the technical solution provided by the present invention, the captured image is decomposed to obtain the top-down perspective feature and the surround-view perspective feature, respectively. The top-down perspective feature will obtain feature information in the horizontal direction, while the surround-view perspective feature map will obtain feature information in the vertical direction (i.e., feature information with height). The top-down perspective feature and the surround-view feature map are then fused into three-dimensional features, greatly reducing the loss of vertical height information of the three-dimensional features. The technical solution provided by the present invention can reconstruct the three-dimensional features based on a single image, reduce the loss of vertical height information, and improve the accuracy of machine vision perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 This is a system diagram to which this application applies.

[0026] Figure 2 It is a schematic diagram of image feature decomposition of a visual perception method provided by an exemplary embodiment of the present application.

[0027] Figure 3 It is a schematic diagram of image feature fusion of a visual perception method provided by another exemplary embodiment of the present application.

[0028] Figure 4 is a flow chart of a visual perception method provided by an exemplary embodiment of the present application;

[0029] Figure 5 This is a flow chart of a visual perception method for obtaining multi-scale features provided by an exemplary embodiment of the present application;

[0030] Figure 6 This is a flowchart of image feature decomposition of a visual perception method provided by an exemplary embodiment of the present application;

[0031] Figure 7 This is a flowchart of an image feature fusion method of visual perception provided by an exemplary embodiment of the present application;

[0032] Figure 8 This is a flow chart of a visual perception method for determining a feature decomposition plane provided by an exemplary embodiment of the present application;

[0033] Figure 9 is a schematic diagram of a visual perception device provided by an exemplary embodiment of the present application;

[0034] Figure 10 is a schematic diagram of an encoding module of a visual perception device provided by an exemplary embodiment of the present application;

[0035] Figure 11 is a schematic diagram of a transformation module of a visual perception device provided by an exemplary embodiment of the present application;

[0036] Figure 12 is a schematic diagram of a fusion module of a visual perception device provided by an exemplary embodiment of the present application;

[0037] Figure 13 is a schematic diagram of a transformation module of a visual perception device provided by an exemplary embodiment of the present application;

[0038] Figure 14 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0039] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0040] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0041] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0042] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0043] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0044] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0045] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0046] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0048] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0049] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0051] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0052] Overview of the Disclosure

[0053] In existing technology, captured images are typically reconstructed into 3D features using an inverse perspective transformation, which assumes the ground is flat. While this transformation works well for objects close to the road, it can severely distort objects at a certain height, such as traffic lights and signs. This results in a loss of vertical information and compromises perception.

[0054] The present disclosure provides a visual perception method, comprising: acquiring multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position; encoding the multiple images respectively to obtain image features corresponding to each image;

[0055] Decompose each image feature by remapping to obtain the top-view feature and the surround-view feature corresponding to each image;

[0056] The features of the top-down view and the surround view are integrated to obtain the three-dimensional spatial features;

[0057] Decode the three-dimensional spatial features to obtain the three-dimensional spatial structure;

[0058] Realize visual perception based on three-dimensional spatial structure.

[0059] The technical solution provided by the present invention decomposes the captured image to obtain the top-down perspective feature and the surround-view perspective feature. The top-down perspective feature captures horizontal feature information, while the surround-view perspective feature captures vertical feature information. The top-down perspective feature and the surround-view perspective feature are then fused into three-dimensional features, significantly reducing the loss of vertical height information in the three-dimensional features. The technical solution provided by the present invention enables reconstruction based on the three-dimensional features of a single image, reducing vertical height information loss and improving the accuracy of machine vision perception.

[0060] Exemplary Systems

[0061] like Figure 1 As shown, an exemplary system used in the process of implementing the visual perception method of the present invention is shown, including multiple image acquisition modules, such as but not limited to a front view image acquisition module ( Figure 1 Front), rear-front image acquisition module ( Figure 1 Rear), and the right front view image acquisition module ( Figure 1 Multiple image acquisition modules can capture images of different perspectives at the same location.

[0062] The collected images are encoded by an encoding module (Encoder), and image features (Feature Mapes) corresponding to each image can be obtained. The encoding module can, for example, use a convolution calculation method to obtain the image features corresponding to each image.

[0063] Each image feature can be remapped to obtain the corresponding surround view feature (Feature Maps in Range View) and the overlook view feature (Feature Maps in Bev View). For the surround view feature, it can provide vertical information, while for the overlook view feature, it can provide horizontal information. The process of remapping each image feature into the range view feature and the overlook view feature is as follows: Figure 2 As shown. Figure 2 In the figure, Current View represents the image captured by the image acquisition module, Range View represents the decomposed surround view feature, and Bev View represents the decomposed top view feature. The intersection of the radial lines connecting the four corners of Current View represents the camera position, and the dotted arrows indicate the decomposition direction.

[0064] The top-view feature and the surround-view feature are then fused through a fusion module to form a three-dimensional spatial feature (Feature Fusion). When forming the three-dimensional spatial feature, the top-view feature is used to determine the horizontal position of each point of the three-dimensional spatial feature, and the surround-view feature is used to determine the vertical position of each point of the three-dimensional spatial feature. The fusion method of the top-view feature and the surround-view feature is as follows: Figure 3 As shown, Range View represents the surround view feature, Bev View represents the top view feature, and Pillar cell represents the point of the three-dimensional space feature formed by the fusion of the corresponding points in the surround view feature and the top view feature.

[0065] The three-dimensional spatial features are then decoded by the decoding module to obtain the three-dimensional spatial structure, which can be used to achieve visual perception.

[0066] The disclosed method

[0067] The embodiment of the present invention provides a visual perception method, such as Figure 4 As shown, including:

[0068] Step 100: Acquire multiple images corresponding to multiple perspectives captured at the same position by a multi-camera system of a vehicle;

[0069] In some embodiments, the multiple images corresponding to multiple perspectives captured at the same location refer to images captured by multiple image capture modules with different perspectives when the vehicle is in the same location. Each image corresponds to a single image capture module. For example, the multiple image capture modules may capture images corresponding to the front perspective, rear perspective, left front perspective, left rear perspective, right front perspective, and right rear perspective, respectively.

[0070] Step 200: Encode the multiple images separately to obtain image features corresponding to each image;

[0071] In some embodiments, encoding an image means performing a convolution operation on the image using a filter for feature extraction to obtain features within the image. Because different filters can correspond to different features, different filters can be used to perform convolution operations on the image during the image encoding process to extract different features within the image. Furthermore, because the image size that can be extracted by the filter is limited, in this embodiment, the image can be scaled to different scales before performing convolution operations using the filter to extract features at different scales.

[0072] Step 300: Decompose each image feature by remapping to obtain the top-view feature and the surround-view feature corresponding to each image;

[0073] In some embodiments, remapping refers to remapping an image from the current image plane to one or more other planes or surfaces based on mathematical relationships. A top-down perspective feature refers to an image feature formed on the horizontal plane where the vehicle resides. A surround perspective feature refers to an image feature formed on an annular surface surrounding the vehicle.

[0074] Step 400: Fusing the top-view features and the surround-view features to obtain a three-dimensional spatial feature.

[0075] In some embodiments, the top-down perspective feature represents the image feature of the plane on which the vehicle resides, and thus represents the distance information of the image feature within that plane. Furthermore, the surround perspective feature represents the image feature of an annular surface perpendicular to the plane on which the vehicle resides, and thus represents the distance information of the image feature within that annular surface. Combining these two features, the position of each point of the three-dimensional spatial feature within three dimensions can be determined.

[0076] Step 500: Decode the three-dimensional spatial features to obtain a three-dimensional spatial structure;

[0077] In some embodiments, the three-dimensional spatial structure includes the shape, position, and posture of each object in the three-dimensional space. Examples include the shape, position, and posture of vehicles, pedestrians, signs, and traffic lights. Decoding the three-dimensional spatial features involves deconvolving the three-dimensional spatial features using multiple filters to restore the three-dimensional spatial features to the three-dimensional spatial structure.

[0078] Step 600: Implement visual perception based on the three-dimensional space structure.

[0079] In some embodiments, a three-dimensional spatial structure refers to a structure formed by constructing image features in three-dimensional space. In three-dimensional space, each target in the image feature, such as other vehicles, pedestrians, signs, and traffic lights, is displayed in three dimensions. Therefore, relative displacement information and relative rotation information between any two targets, as well as the current vehicle and any target, as well as the height information of each target, can be obtained. Based on this relative displacement information, relative rotation information, and the height information of each target, accurate visual perception can be achieved.

[0080] In the technical solution provided by the embodiment of the present invention, by decomposing the collected image, the top-view perspective feature and the surround-view perspective feature are obtained respectively, wherein the top-view perspective feature will obtain feature information in the horizontal direction, and the surround-view perspective feature map will obtain feature information in the vertical direction. The top-view perspective feature and the surround-view perspective feature are then fused into three-dimensional features, which greatly reduces the height information loss of the three-dimensional features in the vertical direction. Through the technical solution provided by the present invention, three-dimensional features can be reconstructed based on the image, reducing the height information loss in the vertical direction and improving the accuracy of machine vision perception. In some preferred embodiments, the horizontal plane can be, for example, a plane parallel to the ground, and the vertical direction can be, for example, a direction perpendicular to the ground.

[0081] In the above Figure 4 Based on the embodiment shown, Figure 5 As shown, in step 200, multiple images are encoded respectively, and the image features corresponding to each image are obtained, including:

[0082] Step 110: scaling each image to obtain an image pyramid corresponding to each image;

[0083] In some embodiments, to obtain features at different scales within an image, it is necessary to scale the image before performing feature extraction. An image pyramid refers to a structure formed by stacking multiple images of different scales. Images with larger scales are ranked lower in the hierarchy, while images with smaller scales are ranked higher in the hierarchy, thus forming a pyramid-like structure.

[0084] Step 120 : performing feature extraction on each image pyramid using multiple feature extraction filters to obtain multi-scale image features corresponding to each image.

[0085] In some embodiments, when performing feature extraction on an image pyramid, multiple feature extraction filters are used to convolve downward layer by layer from the top of the image pyramid to obtain image features at different scales of the image.

[0086] In the above Figure 4 Based on the embodiment shown, Figure 6 As shown, in step 300, each image feature is decomposed by remapping, and the top view angle feature and the surround view angle feature corresponding to each image are obtained, including:

[0087] Step 310 , using the external parameters of each camera in the multi-camera system, calculate a first remapping vector for decomposing each image feature corresponding to each camera into a corresponding top-view feature and a second remapping vector for decomposing each image feature into a corresponding surround view feature;

[0088] In some embodiments, the camera's external parameters refer to the camera's translation and rotation relative to the vehicle, for example, the translation and rotation of the camera center relative to the vehicle's center. The first remapping vector refers to the transformation vector when a point in the image captured by the camera is projected onto the plane where the vehicle is located, and the second remapping vector refers to the transformation vector when a point in the image captured by the camera is projected onto a circular surface perpendicular to the plane where the vehicle is located. In some preferred embodiments, the camera's external parameters represent the camera's translation vector and rotation vector relative to the vehicle. When calculating the first and second remapping vectors, the corresponding mapping vector can be determined by multiplying the translation vector and rotation vector of the coordinate system to be mapped relative to the camera coordinate system, and then multiplying them by the camera's external parameters. For example, for the first remapping vector, it is necessary to multiply the translation vector and rotation vector of the coordinate system corresponding to the top-down perspective feature relative to the camera coordinate system, and then multiply them by the camera's external parameters. Similarly, a similar calculation method can be used to calculate the second remapping vector.

[0089] Step 320 : remap each image feature based on the first remapping vector and the second remapping vector to obtain a top-viewing perspective feature and a surround-viewing perspective feature corresponding to each image feature.

[0090] In some embodiments, remapping each image feature means multiplying each point in each image feature by a first remapping vector to obtain the position of each point in each image feature in the plane where the vehicle is located; and multiplying each point in each image feature by a second remapping vector to obtain the position of each point in each image feature on a circular surface perpendicular to the plane where the vehicle is located.

[0091] In the above Figure 4 Based on the embodiment shown, in step 300, each image feature is decomposed by remapping to obtain the top-viewing perspective feature and the surround-viewing perspective feature corresponding to each image, including:

[0092] The multi-scale image features of each image are decomposed by remapping to obtain the multi-scale bird's-eye view features and multi-scale surround view features of each image.

[0093] In some embodiments, during the feature extraction process, image features at multiple scales are obtained by scaling the image. In order to make the image fusion more accurate, the image features at each scale are decomposed to obtain the top-down perspective features and the surround-view perspective features at each scale.

[0094] In the above Figure 4 Based on the embodiment shown, in step 400, the top-view features and the surround-view features are fused to obtain three-dimensional spatial features including:

[0095] The multi-scale bird's-eye view features and the multi-scale surround view features of each image are fused scale by scale to obtain a three-dimensional spatial feature; wherein the scale-by-scale fusion includes: fusing the bird's-eye view features and surround view features of the same scale one by one.

[0096] In some embodiments, when performing feature fusion, the top-down perspective features at the current scale are fused with the surround-view perspective features at the same scale. After the fusion is complete, the top-down perspective features at the next scale are fused with the surround-view perspective features at the same scale. In some preferred embodiments, the current scale is smaller than the next scale. In other preferred embodiments, the current scale may also be larger than the next scale.

[0097] In the above Figure 4 Based on the embodiment shown, Figure 7 As shown, in step 400, the top-view features and the surround-view features are fused to obtain the three-dimensional spatial features including:

[0098] Step 410: determining the coordinates of each point of the three-dimensional space feature on the horizontal plane coordinate system based on the top-view perspective feature;

[0099] In some embodiments, because the top-down perspective feature is an image feature projected onto a top-down plane, it can clearly represent the position of each point in the three-dimensional spatial feature within the horizontal plane coordinate system. Using the top-down perspective feature to determine the coordinates of each point in the three-dimensional spatial feature in the horizontal plane coordinate system enables accurate positioning and avoids distance loss.

[0100] Step 420, determining the angular coordinates and vertical coordinates of each point of the three-dimensional space feature in the cylindrical coordinate system based on the surround view angle feature;

[0101] In some embodiments, because the surround view feature is an image feature projected onto a curved surface surrounding the vehicle, it can clearly represent the height information of each point in the three-dimensional spatial feature in the angular coordinate system. Using the surround view feature to determine the coordinates of each point in the three-dimensional spatial feature on the cylindrical surface enables accurate positioning and avoids height loss.

[0102] In step 430 , based on the correspondence between the horizontal plane coordinate system and the cylindrical coordinate system, the top-view feature and the surround-view feature are merged to obtain a three-dimensional space feature.

[0103] In some embodiments, the coordinates of each point in the three-dimensional spatial feature are accurately determined based on the top-down and surround-view features. The top-down and surround-view features are then fused to determine the three-dimensional spatial feature, preserving both vertical height information and horizontal information, thereby improving perception accuracy and performance. In some preferred embodiments, the surround-view feature has both height and angle information, while the top-down feature has coordinates defined in two mutually perpendicular directions within a plane. During the fusion process, for example, taking any selected point in the surround-view feature as an example, the height and angle information of that point are determined. A point at the same height in the surround-view feature's vertical coordinate system is connected to that point and extended outward to form an extension line. A perpendicular line is drawn from any point on this extension line to the top-down feature. This perpendicular line will determine a point in the top-down feature. At this point, the point in the top-down feature determined by the perpendicular line corresponds to the point initially selected in the surround-view feature. Fusion of the two points can determine a point in the three-dimensional spatial feature.

[0104] In the above Figure 4 Based on the embodiment shown, Figure 8 As shown, in step 300, each image feature is decomposed by remapping, and the top view angle feature and the surround view angle feature corresponding to each image are obtained, including:

[0105] Step 330 , using the bottom surface of the vehicle as the decomposition plane of the top view feature;

[0106] In some embodiments, since the plane where the vehicle is located is not always in a horizontal plane, for example, when the vehicle is traveling on a slope, the bottom surface of the vehicle is determined as the decomposition plane of the top-view perspective features, so that the decomposition of each image feature is performed along the plane where the bottom surface of the vehicle is located, thereby avoiding a decrease in perception accuracy due to the tilt of the vehicle.

[0107] In step 340 , a cylindrical surface perpendicular to the bottom surface of the vehicle is used as a decomposition plane of the surround view feature.

[0108] In some embodiments, since the boundary of the surround view feature will affect the height information of objects in the image, in order to accurately determine the height information of the objects, a cylindrical surface perpendicular to the bottom surface of the vehicle is used as the decomposition plane of the surround view feature. In some preferred embodiments, the vehicle can be, for example, a car.

[0109] Exemplary devices

[0110] The embodiment of the present invention also provides a visual perception device, such as Figure 9 As shown, including:

[0111] An acquisition module, configured to acquire multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position;

[0112] In some embodiments, the multiple images corresponding to multiple perspectives captured at the same location refer to images captured by multiple image capture modules with different perspectives when the vehicle is in the same location. Each image corresponds to a single image capture module. For example, the multiple image capture modules may capture images corresponding to the front perspective, rear perspective, left front perspective, left rear perspective, right front perspective, and right rear perspective, respectively.

[0113] An encoding module, used to encode multiple images separately to obtain image features corresponding to each image;

[0114] In some embodiments, encoding an image means performing a convolution operation on the image using a filter for feature extraction to obtain features within the image. Because different filters can correspond to different features, different filters can be used to perform convolution operations on the image during the image encoding process to extract different features within the image. Furthermore, because the image size that can be extracted by the filter is limited, in this embodiment, the image can be scaled to different scales before performing convolution operations using the filter to extract features at different scales.

[0115] A transformation module is used to decompose each image feature by remapping to obtain the top-view feature and the surround-view feature corresponding to each image;

[0116] In some embodiments, remapping refers to remapping an image from the current image plane to one or more other planes or surfaces based on mathematical relationships. A top-down perspective feature refers to an image feature formed on the horizontal plane where the vehicle resides. A surround perspective feature refers to an image feature formed on an annular surface surrounding the vehicle.

[0117] A fusion module is used to fuse the top-view features and the surround-view features to obtain three-dimensional spatial features;

[0118] In some embodiments, the top-down perspective feature represents the image feature of the plane on which the vehicle resides, and thus represents the distance information of the image feature within that plane. Furthermore, the surround perspective feature represents the image feature of an annular surface perpendicular to the plane on which the vehicle resides, and thus represents the distance information of the image feature within that annular surface. Combining these two features, the position of each point of the three-dimensional spatial feature within three dimensions can be determined.

[0119] A decoding module is used to decode the three-dimensional spatial features and obtain the three-dimensional spatial structure;

[0120] In some embodiments, the three-dimensional spatial structure includes the shape, position, and posture of each object in the three-dimensional space. Examples include the shape, position, and posture of vehicles, pedestrians, signs, and traffic lights. Decoding the three-dimensional spatial features involves deconvolving the three-dimensional spatial features using multiple filters to restore the three-dimensional spatial features to the three-dimensional spatial structure.

[0121] The perception module is used to realize visual perception based on the three-dimensional space structure.

[0122] In some embodiments, a three-dimensional spatial structure refers to a structure formed by constructing image features in three-dimensional space. In three-dimensional space, each target in the image feature, such as other vehicles, pedestrians, signs, and traffic lights, is displayed in three dimensions. Therefore, the relative displacement and rotation information between any two targets, as well as the relative displacement and rotation information between the current vehicle and any target, and the height information of each target are all available. Based on this relative displacement information, relative rotation information, and the height information of each target, accurate visual perception can be achieved.

[0123] In the technical solution provided by the embodiments of the present invention, the captured image is decomposed to obtain the top-down view feature and the surround view feature. The top-down view feature captures horizontal feature information, while the surround view feature captures vertical feature information. The top-down view feature and the surround view feature are then fused into three-dimensional features, significantly reducing the vertical height information loss of the three-dimensional features. The technical solution provided by the present invention enables 3D reconstruction based on the 3D features of a single image, reducing the vertical height information loss and improving the accuracy of machine vision perception.

[0124] In the above Figure 9 Based on the embodiment shown, Figure 10 As shown, the encoding module includes:

[0125] A scaling unit, used to scale each image to obtain an image pyramid corresponding to each image;

[0126] In some embodiments, to obtain features at different scales within an image, it is necessary to scale the image before performing feature extraction. An image pyramid refers to a structure formed by stacking multiple images of different scales. Images with larger scales are ranked lower in the hierarchy, while images with smaller scales are ranked higher in the hierarchy, thus forming a pyramid-like structure.

[0127] The extraction unit is used to perform feature extraction on each image pyramid using multiple feature extraction filters to obtain multi-scale image features corresponding to each image.

[0128] In some embodiments, when performing feature extraction on an image pyramid, multiple feature extraction filters are used to convolve downward layer by layer from the top of the image pyramid to obtain image features at different scales of the image.

[0129] In the above Figure 9 Based on the embodiment shown, Figure 11 As shown, the transformation module includes:

[0130] a calculation unit, configured to calculate, by using the external parameters of each camera in the multi-camera system, a first remapping vector for decomposing each image feature corresponding to each camera into a corresponding top-view feature and a second remapping vector for decomposing each image feature into a corresponding surround-view feature;

[0131] In some embodiments, the camera's external parameters refer to the camera's translation and rotation relative to the vehicle, for example, the translation and rotation of the camera center relative to the vehicle's center. The first remapping vector refers to the transformation vector when a point in the image captured by the camera is projected onto the plane where the vehicle is located, and the second remapping vector refers to the transformation vector when a point in the image captured by the camera is projected onto a circular surface perpendicular to the plane where the vehicle is located. In some preferred embodiments, the camera's external parameters represent the camera's translation vector and rotation vector relative to the vehicle. When calculating the first and second remapping vectors, the corresponding mapping vector can be determined by multiplying the translation vector and rotation vector of the coordinate system to be mapped relative to the camera coordinate system, and then multiplying them by the camera's external parameters. For example, for the first remapping vector, it is necessary to multiply the translation vector and rotation vector of the coordinate system corresponding to the top-down perspective feature relative to the camera coordinate system, and then multiply them by the camera's external parameters. Similarly, a similar calculation method can be used to calculate the second remapping vector.

[0132] The mapping unit is configured to remap each image feature based on the first remapping vector and the second remapping vector to obtain a top-viewing perspective feature and a surround-viewing perspective feature corresponding to each image feature.

[0133] In some embodiments, remapping each image feature means multiplying each point in each image feature by a first remapping vector to obtain the position of each point in each image feature in the plane where the vehicle is located; and multiplying each point in each image feature by a second remapping vector to obtain the position of each point in each image feature on a circular surface perpendicular to the plane where the vehicle is located.

[0134] In the above Figure 9 Based on the embodiment shown, the transformation module is further configured to decompose the multi-scale image features of each image by remapping to obtain multi-scale top-view features and multi-scale surround-view features of each image.

[0135] In some embodiments, during the feature extraction process, image features at multiple scales are obtained by scaling the image. In order to make the image fusion more accurate, the image features at each scale are decomposed to obtain the top-down perspective features and the surround-view perspective features at each scale.

[0136] In the above Figure 9 Based on the embodiment shown, the fusion module is further used to fuse the multi-scale bird's-eye view features and the multi-scale surround view features of each image scale by scale to obtain three-dimensional spatial features; wherein, the scale-by-scale fusion includes: fusing the bird's-eye view features and surround view features with the same scale one by one.

[0137] In some embodiments, when performing feature fusion, the top-down perspective features at the current scale are fused with the surround-view perspective features at the same scale. After the fusion is complete, the top-down perspective features at the next scale are fused with the surround-view perspective features at the same scale. In some preferred embodiments, the current scale is smaller than the next scale. In other preferred embodiments, the current scale may also be larger than the next scale.

[0138] In the above Figure 9 Based on the embodiment shown, Figure 12 As shown, the fusion module includes:

[0139] A horizontal calculation unit, configured to determine the coordinates of each point of the three-dimensional spatial feature on a horizontal plane coordinate system based on the top-down perspective feature;

[0140] In some embodiments, because the top-down perspective feature is an image feature projected onto a top-down plane, it can clearly represent the position of each point in the three-dimensional spatial feature within the horizontal plane coordinate system. Using the top-down perspective feature to determine the coordinates of each point in the three-dimensional spatial feature in the horizontal plane coordinate system enables accurate positioning and avoids distance loss.

[0141] A vertical calculation unit, used to determine the angular coordinates and vertical coordinates of each point of the three-dimensional space feature in the cylindrical coordinate system based on the surround view angle feature;

[0142] In some embodiments, because the surround view feature is an image feature projected onto a curved surface surrounding the vehicle, it can clearly represent the height information of each point in the three-dimensional spatial feature in the angular coordinate system. Using the surround view feature to determine the coordinates of each point in the three-dimensional spatial feature on the cylindrical surface enables accurate positioning and avoids height loss.

[0143] The fusion calculation unit is used to fuse the top-view perspective feature and the surround-view perspective feature according to the correspondence between the horizontal plane coordinate system and the cylindrical coordinate system to obtain the three-dimensional space feature.

[0144] In some embodiments, the coordinates of each point in the three-dimensional spatial feature are accurately determined based on the top-down and surround-view features. The three-dimensional spatial feature determined by fusing the top-down and surround-view features can preserve both vertical height information and horizontal information, thereby improving perception accuracy and performance. In some preferred embodiments, the surround-view feature has both height and angle information, while the top-down feature has coordinates defined in two mutually perpendicular directions within a plane. During the fusion process, for example, taking any selected point in the surround-view feature as an example, the height and angle information of that point are determined. A line is drawn from any point on the extended line perpendicular to the top-down feature, and the line is then drawn perpendicular to the top-down feature. This line then defines a point in the top-down feature. At this point, the point in the top-down feature defined by the perpendicular line corresponds to the point initially selected in the surround-view feature. Fusing the two points together can determine a point in the three-dimensional spatial feature.

[0145] In the above Figure 9 Based on the embodiment shown, Figure 13 As shown, the transformation module includes:

[0146] A first determining unit is configured to use the bottom surface of the vehicle as a decomposition plane of the top-view feature;

[0147] In some embodiments, since the plane where the vehicle is located is not always in a horizontal plane, for example, when the vehicle is traveling on a slope, the bottom surface of the vehicle is determined as the decomposition plane of the top-view perspective features, so that the decomposition of each image feature is performed along the plane where the bottom surface of the vehicle is located, thereby avoiding a decrease in perception accuracy due to the tilt of the vehicle.

[0148] The second determining unit is configured to use a cylindrical surface perpendicular to the bottom surface of the vehicle as a decomposition plane of the surround view angle feature.

[0149] In some embodiments, since the boundary of the surround view feature will affect the height information of objects in the image, in order to accurately determine the height information of the objects, a cylindrical surface perpendicular to the bottom surface of the vehicle is used as the decomposition plane of the surround view feature. In some preferred embodiments, the vehicle can be, for example, a car.

[0150] Exemplary electronic devices

[0151] Below, reference Figure 14The electronic device according to the embodiment of the present application is described. The electronic device can be either or both of the first device 100 and the second device 200, or a standalone device independent of them, and the standalone device can communicate with the first device and the second device to receive the collected input signals from them.

[0152] Figure 14 The figure shows a block diagram of an electronic device according to an embodiment of the present application.

[0153] like Figure 14 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0154] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0155] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the visual perception method of each embodiment of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0156] In one example, the electronic device 10 may further include an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0157] For example, when the electronic device is the first device 100 or the second device 200, the input device 13 may be the aforementioned camera for capturing images. When the electronic device is a standalone device, the input device 13 may be a communication network connector for receiving captured input signals from the first device 100 and the second device 200.

[0158] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0159] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0160] Of course, to simplify, Figure 14 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 10 may further include any other appropriate components according to specific application scenarios.

[0161] Exemplary computer program products and computer-readable storage media

[0162] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the visual perception method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0163] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0164] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the visual perception method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0165] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0166] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0167] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0168] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0169] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0170] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A visual perception method, comprising: Acquire multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position; Encoding the multiple images respectively to obtain image features corresponding to the images; Decomposing each of the image features by remapping to obtain a top-view perspective feature and a surround-view perspective feature corresponding to each of the images; Fusing the top-view features and the surround-view features to obtain three-dimensional spatial features; Decoding the three-dimensional spatial features to obtain a three-dimensional spatial structure; achieving visual perception according to the three-dimensional spatial structure; The three-dimensional space features obtained by fusing the top-view features and the surround-view features include: Determining the coordinates of each point of the three-dimensional space feature on a horizontal plane coordinate system based on the top-view perspective feature; Determining the angular coordinates and vertical coordinates of each point of the three-dimensional space feature in a cylindrical coordinate system based on the surround viewing angle feature; According to the correspondence between the horizontal plane coordinate system and the cylindrical coordinate system, the top-view feature and the surround-view feature are fused to obtain the three-dimensional space feature.

2. The method according to claim 1, wherein Encoding the multiple images separately to obtain image features corresponding to the images includes: scaling each of the images to obtain an image pyramid corresponding to each of the images; A plurality of feature extraction filters are used to perform feature extraction on each of the image pyramids to obtain multi-scale image features corresponding to each of the images.

3. The method according to claim 2, wherein: Decomposing each of the image features by remapping, the top-view angle features and the surround-view angle features corresponding to each of the images are obtained, including: Utilizing the external parameters of each camera in the multi-camera system, calculating a first remapping vector for decomposing each image feature corresponding to each camera into a corresponding top-view feature and a second remapping vector for decomposing each image feature into a corresponding surround-view feature; Based on the first remapping vector and the second remapping vector, each of the image features is remapped to obtain the top-view perspective feature and the surround-view perspective feature corresponding to each of the image features.

4. The method according to claim 2, wherein: Decomposing each of the image features by remapping, the top-view angle features and the surround-view angle features corresponding to each of the images are obtained, including: The multi-scale image features of each of the images are decomposed in a remapping manner to obtain multi-scale top-view features and multi-scale surround-view features of each of the images.

5. The method according to claim 4, wherein The three-dimensional spatial features obtained by fusing the top-view features and the surround-view features include: The multi-scale top-view features and the multi-scale surround-view features of each image are fused scale by scale to obtain a three-dimensional spatial feature; wherein the scale-by-scale fusion includes: fusing the top-view features and surround-view features with the same scale one by one.

6. The method according to claim 1, wherein Decomposing each of the image features by remapping, the top-view angle features and the surround-view angle features corresponding to each of the images are obtained, including: The bottom surface of the vehicle is used as the decomposition plane of the top-view feature; A cylindrical surface perpendicular to the bottom surface of the vehicle is used as a decomposition plane of the surround view feature.

7. A visual perception device comprising: An acquisition module, configured to acquire multiple images corresponding to multiple perspectives captured by a multi-camera system of a vehicle at the same position; An encoding module, configured to encode the plurality of images respectively to obtain image features corresponding to the images; a transformation module, configured to decompose each of the image features by remapping to obtain a top-view feature and a surround-view feature corresponding to each of the images; A fusion module, configured to fuse the top-view features and the surround-view features to obtain a three-dimensional spatial feature; A decoding module, configured to decode the three-dimensional spatial features to obtain a three-dimensional spatial structure; A perception module, configured to realize visual perception based on the three-dimensional spatial structure; The fusion module includes: A horizontal calculation unit, configured to determine the coordinates of each point of the three-dimensional spatial feature on a horizontal plane coordinate system based on the top-down perspective feature; A vertical calculation unit, used to determine the angular coordinates and vertical coordinates of each point of the three-dimensional space feature in the cylindrical coordinate system based on the surround view angle feature; The fusion calculation unit is used to fuse the top-view perspective feature and the surround-view perspective feature according to the correspondence between the horizontal plane coordinate system and the cylindrical coordinate system to obtain the three-dimensional space feature.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the visual perception method according to any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the visual perception method described in any one of claims 1-6.

Images