A method, device and computer storage medium for determining posture based on a bird's-eye view

Abstract

The present invention discloses a method, device and computer storage medium for determining posture based on a bird's-eye view, which belongs to the field of visual positioning technology, and aims to solve the technical problem that the current method requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenes. The method comprises the following steps: obtaining a bird's-eye view and an original image of the same scene. Masking the moving target in the bird's-eye view to obtain a masked bird's-eye view. Processing the masked bird's-eye view using a deep learning network to obtain a first feature. Processing the original image using a deep posture estimation network to obtain a second feature. Fusion of the first feature and the second feature to obtain the scene posture.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning technology, and in particular to a method, device and computer storage medium for determining a posture based on a bird's-eye view. Background Art

[0002] As visual localization technology has entered a stage of robust perception, people have devoted themselves to studying methods to perform high-level scene perception and understanding to increase the robustness of visual localization tasks.

[0003] Most current methods mainly use semantic understanding to link semantic concepts (such as object classification, material composition, etc.) with the geometric structure of the environment. However, this method often requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a method, device and computer storage medium for determining posture based on a bird's-eye view, so as to solve the technical problem that the current method requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios.

[0005] In a first aspect, the present invention provides a method for determining a pose based on a bird's-eye view, comprising the following steps: obtaining a bird's-eye view and an original image of the same scene. Masking a moving target in the bird's-eye view to obtain a masked bird's-eye view. Processing the masked bird's-eye view using a deep learning network to obtain a first feature. Processing the original image using a deep pose estimation network to obtain a second feature. Fusion of the first feature and the second feature to obtain a scene pose.

[0006] Compared with the prior art, the method for determining the pose based on the bird's-eye view provided by the present invention first obtains the bird's-eye view and the original image of the same scene; then the moving target in the bird's-eye view is masked to obtain the masked bird's-eye view. The masked bird's-eye view is processed by a deep learning network to obtain the first feature. The original image is then processed by a deep pose estimation network to obtain the second feature. Finally, the first feature and the second feature are fused to obtain the scene pose. In practice, since the original image is an image taken by a camera, the size and degree of movement of different objects will vary with the depth of the object from the camera (which can also be understood as the different perspectives of the object), and the degree of movement of the object is what we need to pay attention to, which also leads to a certain error in the final result. Since the bird's-eye view is a stereoscopic image drawn from a certain point at a high altitude using a high viewpoint perspective method based on the perspective principle, the adverse effect of perspective changes on the extraction of the actual target entity position relationship can be solved. Therefore, the present invention can improve the ability to understand the real scene. Based on this, the present invention can not only establish more robust features for scene pose determination, but also distinguish different states of dynamic foreground and static background to a certain extent, that is to say, this understanding can enhance robustness and counter the ambiguity of dynamic and static objects to a certain extent. Therefore, the scene pose obtained by fusing the first feature and the second feature has high robustness, and the present invention is based on feature extraction to obtain the scene pose. It solves the technical problem that the current methods of the prior art require a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenes.

[0007] In a second aspect, the present invention also provides a posture determination device based on a bird's-eye view, comprising a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement any of the above-mentioned posture determination methods based on a bird's-eye view.

[0008] In a third aspect, the present invention further provides a computer storage medium, in which instructions are stored, and when the instructions are executed, a method for determining a posture based on a bird's-eye view is implemented.

[0009] Compared with the prior art, the beneficial effects of the second and third aspects of the present invention are the same as the beneficial effects of the posture determination method based on the bird's-eye view described in the above technical solution, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 A flowchart of a method for determining a posture based on a bird's-eye view provided by an embodiment of the present invention;

[0012] Figure 2 A 3D radar point cloud image provided by an embodiment of the present invention;

[0013] Figure 3 An RGB original image provided by an embodiment of the present invention;

[0014] Figure 4 A bird's-eye view with target detection results provided by an embodiment of the present invention;

[0015] Figure 5 A schematic diagram of the locations of various entities in a bird's-eye view provided by an embodiment of the present invention;

[0016] Figure 6 A structural block diagram of a device for determining the position and posture of a camera provided in an embodiment of the present invention;

[0017] Figure 7 A schematic diagram of the hardware structure of a device for determining the position and posture of a camera provided in an embodiment of the present invention;

[0018] Figure 8 A schematic diagram of the structure of a chip provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0020] Various schematic diagrams of embodiments of the present invention are shown in the accompanying drawings, which are not drawn to scale. For the purpose of clarity, some details are magnified and some details may be omitted. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0021] In the following, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0022] In the present invention, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium.

[0023] In related technologies, as visual localization technology has entered a stage of robust perception, people have been committed to studying methods to perform high-level scene perception and understanding to increase the robustness of visual localization tasks.

[0024] Most current methods mainly use semantic understanding to link semantic concepts (such as object classification, material composition, etc.) with the geometric structure of the environment. However, this method often requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios.

[0025] Based on this, an embodiment of the present invention discloses a method for determining a pose based on a bird's-eye view, which is used to solve the technical problem that the current method requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios.

[0026] The method for determining a posture based on a bird's-eye view in an embodiment of the present invention is applied to a posture determining device based on a bird's-eye view. Figure 1 ,The pose determination method based on the bird's-eye view includes the following steps:

[0027] S101, obtaining a bird's-eye view and an original image of the same scene.

[0028] In practice, there are many ways to obtain a bird's-eye view. It can be obtained by performing a certain projection on the radar point cloud data. It can also be obtained from an electronic map. Among them, Figure 2 A schematic diagram of a bird's-eye view obtained from 3D radar point cloud data is shown.

[0029] In practice, when the bird's-eye view image is a point cloud image after coordinate transformation, the size parameters of the bird's-eye view image should be the same as the size parameters of the original image.

[0030] When the data type of the bird's-eye view is point cloud data, obtaining the bird's-eye view includes:

[0031] First, the original point cloud data is obtained, wherein the original point cloud data can be obtained from the radar point cloud data.

[0032] Then, the data in the first quadrant of the original point cloud data is extracted as the new point cloud data. The X-axis of the new point cloud data is used as the Y-axis of the initial bird's-eye view, and the Y-axis of the new point cloud data is used as the X-axis of the initial bird's-eye view.

[0033] Finally, the initial bird's-eye view image is processed according to the size information and image content of the original image to obtain a bird's-eye view image. The initial bird's-eye view image is processed according to the size information and image content of the original image to make the size information of the obtained bird's-eye view image the same as the size information of the original image, and the image content of the bird's-eye view image represents the same scene as the image content of the original image.

[0034] There are many ways to obtain the original image, for example, through a camera. Specifically, the camera can be used to collect images in a three-dimensional scene to obtain the original image of the scene. For example, Figure 3 An original image is shown, which is an RGB image.

[0035] It is worth noting that the embodiment of the present invention uses a combination of a bird's-eye view and an original image to determine the scene pose, so the above-mentioned bird's-eye view and the original image are the same scene image.

[0036] S102, performing mask processing on the moving target in the bird's-eye view to obtain a masked bird's-eye view.

[0037] In practice, before performing mask processing on moving targets in the bird's-eye view, it is necessary to detect the moving targets in the bird's-eye view. In practice, a target detection network can be used to perform target detection on the bird's-eye view to obtain a bird's-eye view with target detection results.

[0038] Specifically, target detection is to detect targets in an image and mark the position of the target in the image. The embodiment of the present invention can adopt any existing target detection network, such as: R-CNN network, whose main idea is to first generate a series of sparse candidate boxes through heuristic methods (selective search) or CNN network (RPN), and then classify and regress these candidate boxes. Another example: Yolo network and SSD network, whose main idea is to evenly perform dense sampling at different positions of the picture, different scales and aspect ratios can be used for sampling, and then use CNN to extract features and directly perform classification and regression. The whole process only requires one step, so its advantage is fast speed.

[0039] After processing the bird's-eye view image using the above target detection network, a bird's-eye view image with target detection results is obtained. Figure 4 A bird's-eye view of target detection results is shown. It can be understood that any target detection framework with good performance can obtain the following Figure 4 A bird's-eye view with targets is shown, where the higher the target detection accuracy, the better the final model performance.

[0040] Figure 5 (a) shows the positions of various entities in the bird’s-eye view of the nth frame. Figure 5(b) shows the positions of various entities in the bird's-eye view of the n+ith frame, where i=1, 2, 3. Figure 5 (b) The entity in the dotted box is the entity that moves from the nth frame bird's-eye view to the n+ith frame bird's-eye view, that is, the moving target.

[0041] In order to emphasize the importance of the position of each entity in the bird's-eye view and reduce the noise caused by factors such as discrete entity shapes, the embodiment of the present invention masks the targets in the bird's-eye view with target detection results. A mask is a template for an image filter. For example, when extracting roads, rivers, or houses, an n*n matrix is ​​used to filter the pixels of the image, and then the objects or signs we need are highlighted. This matrix is ​​a mask.

[0042] Image masks are mainly used for: ① Extracting the region of interest, multiplying the pre-made region of interest mask with the image to be processed to obtain the region of interest image, the image value in the region of interest remains unchanged, and the image value outside the region is 0. ② Shielding effect, using a mask to shield certain areas on the image so that they do not participate in the processing or the calculation of processing parameters, or only processing or statistics are performed on the shielded area. ③ Structural feature extraction, using similarity variables or image matching methods to detect and extract structural features in the image that are similar to the mask. ④ Production of special shape images. It can be seen that the image mask in the embodiment of the present invention mainly plays a shielding role, which is used to weaken the weight of moving targets (dynamic objects) and increase the weight of static objects.

[0043] As a specific implementation, masking the moving target in the bird's-eye view with the target detection result includes: resetting the pixel value of the moving target area in the bird's-eye view using a mask matrix (also called a kernel), so as to subsequently redistribute the weights of the various areas in the bird's-eye view and reduce the weight of the moving area in the bird's-eye view.

[0044] S103: Use a deep learning network to process the mask bird's-eye view image to obtain a first feature.

[0045] Among them, deep learning is a type of machine learning, and machine learning is the only way to achieve artificial intelligence. The concept of deep learning originated from the study of artificial neural networks. Multilayer perceptrons with multiple hidden layers are a type of deep learning structure. Deep learning combines low-level features to form more abstract high-level representations of attribute categories or features to discover distributed feature representations of data. The motivation for studying deep learning is to establish a neural network that simulates the human brain for analysis and learning. It imitates the mechanism of the human brain to interpret data, such as images, sounds, and text.

[0046] The deep learning network is a network built based on deep learning. The deep learning network in the embodiment of the present invention can be any supervised deep learning network or any unsupervised deep learning network. For example, the supervised learning network can be a posenet network, a deepvo network, etc. For example, the unsupervised learning network can be an sfmlearner network, etc.

[0047] The embodiment of the present invention inputs the mask bird's-eye view into the deep learning network, and the deep learning network adjusts the weights of each target area in the mask bird's-eye view to reduce or even remove the weights of dynamic targets. Therefore, the influence of dynamic targets on the determination of posture can be reduced to a certain extent, the robustness of the determination of scene posture can be improved, and the noise such as edge sharpening caused by directly removing dynamic objects can be avoided. Therefore, the features input from the deep learning network are features that are not affected by the change of viewing angle, that is, static features.

[0048] In practice, the embodiments of the present invention can extract the relationship features between targets in the bird's-eye view with target detection results in different scenes through a deep learning network. Such relationship features reflect the status of each target in different scenes, thereby enhancing the ability to understand the scene.

[0049] The embodiment of the present invention establishes the relationship features between different target entities in the picture, annotates the foreground targets such as cars and people extracted by the target detection framework on the bird's-eye view, and inputs the bird's-eye view sequence with the target detection results into the learning model, and uses the classic deep learning network to extract the features that are not affected by the perspective change. This feature reflects the positional relationship between entities in different scenes and enhances the understanding of the overall state of the scene in the 3D scene. At the same time, the change in the positional relationship of entities during the scene change also reflects the motion state of each entity. While improving the scene understanding ability, it can also remove the features of dynamic entities with large changes in motion state to a certain extent, thereby improving the robustness of the present invention in dynamic scenes.

[0050] S104: Process the original image using a deep pose estimation network to obtain a second feature.

[0051] At present, pose estimation has very important applications in the fields of robotics, automation and machine vision. Especially in the field of robotics, it is very important to obtain the six-dimensional pose of an object accurately and quickly for the robot to operate the object. This is even more true in industrial production. Accurately measuring the pose of accessories can enable industrial robots to grasp objects in a specified posture and align them for installation, which is of great significance to improving industrial production efficiency.

[0052] The pose estimation methods include: the traditional pose estimation methods mainly include methods based on single purpose, methods based on dual purpose, methods based on multi-purpose, methods based on scanning laser radar, methods based on non-scanning laser radar, and the pose estimation methods that have developed rapidly in recent years include methods based on SLAM (Simultaneous Localization And Mapping, synchronous positioning and map construction) and multi-sensor fusion methods. The deep pose estimation network used in the embodiment of the present invention can be a pose estimation network constructed using the above pose estimation methods.

[0053] Specifically, using a deep pose estimation network to process the original image to obtain a second feature includes: inputting the original image into the deep pose estimation network, and using the deep pose estimation network to process the original image to obtain the second feature. The second feature is a pose feature in the original image. Exemplarily, the pose feature can be three position parameters and three pose parameters of a target obtained in a specific coordinate system, and the specific coordinate system can be a world coordinate system, an object coordinate system, or a camera coordinate system.

[0054] S105, fusing the first feature and the second feature to obtain the scene pose.

[0055] In the embodiment of the present invention, fusing the first feature and the second feature to obtain the scene pose may include:

[0056] S1051, convolve the first feature to obtain a first convolution feature.

[0057] Convolving the first feature means performing mathematical operations on the first feature and the corresponding convolution kernel to extract certain specified features. It can be understood that different convolution kernels have different specified features to be extracted here, and thus, the effects are also different. It can be understood that the role of the convolution kernel is to extract features. The larger the convolution kernel size, the larger the receptive field, and of course, more parameters. In an image, since there is local correlation in the spatial domain of the image, the convolution process can also be considered as an extraction of local correlation.

[0058] In order to finally obtain a better information combination, the convolution used in the present invention is 1*1 convolution, which can also be understood as a fully connected layer. Among them, 1*1 convolution is used to achieve cross-channel interaction and information integration. The number of convolution kernel channels is reduced and increased in dimension. For a single-channel feature map, a single-kernel convolution is multiplied by a parameter, while in general, multi-kernel convolution is multi-channel to achieve a linear combination of multiple features.

[0059] S1052, convolve the second feature to obtain a second convolution feature.

[0060] Convolving the second feature means performing mathematical operations on the second feature and the corresponding convolution kernel to extract certain specified features. It can be understood that different convolution kernels will result in different specified features being extracted here, and thus, the effects will also be different.

[0061] Similarly, in order to reduce the convolution kernel parameters, the convolution operation on the second feature can be performed using a 1*1 convolution. The 1*1 convolution is used to achieve cross-channel interaction and information integration. The number of convolution kernel channels is reduced and increased.

[0062] S1053: Fuse the first convolution feature and the second convolution feature to obtain the scene pose.

[0063] In the embodiment of the present invention, the dimensions of the last dimensions of the first convolution feature and the second convolution feature obtained by step S1051 and step S1052 are the same.

[0064] The first convolution feature and the second convolution feature are concatenated, and the average value of the concatenated features is calculated. Specifically, the average value of the concatenated features in one dimension and two dimensions is calculated to obtain the scene pose.

[0065] Among them, the first convolution feature is a feature extracted based on a bird's-eye view, and the second convolution feature is a feature extracted based on an original image. Therefore, the embodiment of the present invention is based on a bird's-eye view and an original image from the same scene, and uses a deep learning network to obtain the final scene pose. Based on this, the method provided in the embodiment of the present invention can improve the ability to understand 3D real scenes, and improve robustness and accuracy. Specifically, after target extraction and target masking of the bird's-eye view, the features of the bird's-eye view are extracted using a deep learning network. This feature can model the relationship features between target entities in different scenes to a certain extent. At the same time, the bird's-eye view solves the adverse effects of perspective changes on the extraction of the actual target entity position relationship, improves the ability to understand 3D real scenes, and at the same time, this understanding of the scene can not only establish more robust features for pose estimation, but also distinguish different states of dynamic foreground and static background to a certain extent, that is, this understanding can enhance the robustness of vo (Visual odometry, monocular visual odometer) and the ambiguity of combating dynamic and static objects to a certain extent.

[0066] To expand, first, the embodiment of the present invention extracts features (first features) from a bird's-eye view sequence with a target mask, and fuses the features with the original image features (second features), thereby enhancing the scene understanding capability, increasing the robustness features, and reducing the influence of dynamic points on the system pose estimation to a certain extent. Furthermore, in the embodiment of the present invention, robust bird's-eye view features that are not affected by changes in viewing angles are added, and the relationship features between different target entities in the bird's-eye view are extracted to deeply understand the scene, and this relationship feature enables the system to understand the motion state of different targets in different scenes. Driven by a large amount of data, the weight of dynamic points will gradually decrease during training, that is, the influence of dynamic points on the system is reduced, and the robustness of the system is further improved, which also avoids directly removing noise such as edge sharpening caused by dynamic objects. Finally, the embodiment of the present invention can provide more robust features and reduce the influence of dynamic points by enhancing the scene understanding capability, and ultimately achieve an improvement in visual positioning accuracy.

[0067] In practice, when the dimension of the scene posture obtained by fusing the first convolution feature and the second convolution feature is different from the preset dimension, the dimension of the scene posture can be adjusted to make the dimension of the scene posture the same as the preset dimension.

[0068] The preset dimension can be set according to the output dimension of the deep learning network. For example, when the deep learning network is a supervised deep learning model Posenet, the output dimension is a 1*6-dimensional matrix. When the deep learning network is an unsupervised model SFMlearner, the scene pose of three consecutive images needs to be obtained at the same time. The purpose is to reproject the t-1 frame and t+1 frame images and their scene poses to obtain the t-frame reconstructed image and minimize the reprojection error. Therefore, the output result dimension is 3*6.

[0069] In an embodiment of the present invention, the dimension of the scene pose may be adjusted in the fully connected layer so that the dimension of the scene pose is the same as the preset dimension.

[0070] Among them, fully connected layers (FC) play the role of "classifier" in the entire convolutional neural network. In actual use, fully connected layers can be implemented by convolution operations: fully connected layers with fully connected previous layers can be converted into convolutions with a convolution kernel of 1*1; and fully connected layers with convolutional previous layers can be converted into global convolutions with a convolution kernel of h*w, where h and w are the height and width of the convolution result of the previous layer, respectively.

[0071] Taking VGG-16 as an example, for an input of 224x224x3, the last layer of convolution can obtain an output of 7x7x512. If the next layer is a FC layer with 4096 neurons, a global convolution with a convolution kernel of 7x7x512x4096 can be used to implement this fully connected operation process. After this convolution operation, the output is 1x1x4096, that is, the output after the fully connected layer.

[0072] Therefore, after passing through the fully connected layer, the dimension of the scene pose can be adjusted to the preset dimension.

[0073] In an embodiment of the present invention, the fused features (scene pose) not only enhance the ability to understand the scene, but also improve the robustness of visual positioning in a dynamic environment. Since in a dynamic scene, the degree of motion change of the static objects and dynamic objects extracted by the relational features is different, and the static objects are more helpful for estimating the pose, the deep learning network's attention to static features is continuously strengthened during the training process. Therefore, under the action of large-scale data-driven and powerful self-optimization of the deep learning network, with the increase of training data and the increase of training times, under the action of the fully connected layer, the dynamic features and static features are better distinguished, and the weights assigned to the dynamic features are gradually reduced, so that the system can still maintain good performance in dynamic scenes.

[0074] In the case of dividing each functional module into corresponding functional modules, Figure 6 FIG. 2 shows a structural block diagram of a device for determining a posture based on a bird's-eye view provided by an embodiment of the present invention. Figure 6 As shown, the bird's-eye view-based posture determination device 70 includes: a communication module 701 and a processing module 702 .

[0075] The communication module 701 is used to obtain images captured by the camera.

[0076] The processing module 702 is used to support the bird's-eye view-based posture determination device to execute steps 101 to 103 in the above embodiment.

[0077] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0078] In some possible implementations, the above-mentioned bird's-eye view-based posture determination device may further include a storage module 703 for storing program codes and data of the base station.

[0079] Among them, the processing module can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of the present invention. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module can be a transceiver, a transceiver circuit or a communication interface, and the like. The storage module can be a memory.

[0080] When the processing module is a processor, the communication module is a communication interface, and the storage module is a memory, the apparatus for determining a posture based on a bird's-eye view according to an embodiment of the present invention may be: Figure 7 A bird's-eye view-based pose determination device is shown.

[0081] Figure 7 FIG. 1 is a schematic diagram showing the hardware structure of a device for determining a posture based on a bird's-eye view provided by an embodiment of the present invention. Figure 7 As shown, the bird's-eye view-based posture determination device 80 includes a processor 801 and a communication interface 802 .

[0082] like Figure 7 As shown, the processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. The communication interface may be one or more. The communication interface may use any transceiver-like device for communicating with other devices or communication networks.

[0083] like Figure 7 As shown, the above-mentioned bird's-eye view-based posture determination device may further include a communication line 803. The communication line may include a path to transmit information between the above-mentioned components.

[0084] Optional, such as Figure 7As shown, the bird's-eye view-based posture determination device may also include a memory 804. The memory is used to store computer-executable instructions for executing the solution of the present invention, and the execution is controlled by the processor. The processor is used to execute the computer-executable instructions stored in the memory, thereby implementing the method provided by the embodiment of the present invention.

[0085] like Figure 7 As shown, the above-mentioned memory can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can exist independently and be connected to the processor through a communication line. The memory can also be integrated with the processor.

[0086] Optionally, the computer-executable instructions in the embodiment of the present invention may also be referred to as application program codes, which is not specifically limited in the embodiment of the present invention.

[0087] In a specific implementation, as an example, Figure 7 As shown, the processor 801 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 in.

[0088] In a specific implementation, as an example, Figure 7 As shown, the device for determining the pose based on the bird's-eye view may include multiple processors, such as Figure 7 Each of these processors may be a single-core processor or a multi-core processor.

[0089] Figure 8 Schematic diagram of the structure of the chip provided by the embodiment of the present invention. Figure 8 As shown, the chip 90 includes one or more (including two) processors 801 and a communication interface 802 .

[0090] Optional, such as Figure 8As shown, the chip also includes a memory 804, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0091] In some embodiments, Figure 8 As shown, the memory stores the following elements, execution modules or data structures, or their subsets, or their extended sets.

[0092] In the embodiment of the present invention, Figure 8 As shown, the corresponding operation is performed by calling the operation instruction stored in the memory (the operation instruction may be stored in the operating system).

[0093] like Figure 8 As shown, the processor controls the processing operations of any one of the bird's-eye view based posture determination devices, and the processor can also be called a central processing unit (CPU).

[0094] like Figure 8 As shown, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 8 Various buses are labeled as bus system 805 .

[0095] like Figure 8As shown, the method disclosed in the above embodiment of the present invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0096] In one possible implementation, Figure 8 As shown, the communication interface is used to obtain the image captured by the camera. The processor is used to execute Figure 1 Steps 101 to 103 of the method for determining a pose based on a bird's-eye view in the illustrated embodiment.

[0097] On the one hand, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed, the functions performed by the bird's-eye view-based posture determination device in the above-mentioned embodiment are implemented.

[0098] On the one hand, a chip is provided, which is applied to a posture determination device based on a bird's-eye view. The chip includes at least one processor and a communication interface. The communication interface is coupled to at least one processor, and the processor is used to run instructions to implement the functions performed by the posture determination device based on a bird's-eye view in the above embodiments.

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present invention is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0100] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0101] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0102] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for determining a posture based on a bird's-eye view, characterized in that: The method comprises the following steps: Get a bird's-eye view and original image of the same scene; Performing mask processing on the moving target in the bird's-eye view to obtain a masked bird's-eye view; Processing the mask bird's-eye view image using a deep learning network to obtain a first feature; Processing the original image using a deep pose estimation network to obtain a second feature; Fusing the first feature and the second feature to obtain a scene pose; The data type of the bird's-eye view is point cloud data; the step of obtaining a bird's-eye view of the same scene includes: Get the original point cloud data; Extracting data in the first quadrant of the two-dimensional coordinate system where the original point cloud data is located as new point cloud data; Using the X-axis of the new point cloud data as the Y-axis of the initial bird's-eye view, and the Y-axis of the new point cloud data as the X-axis of the initial bird's-eye view; The initial bird's-eye view image is processed according to the size information and image content of the original image to obtain the bird's-eye view image.

2. The method for determining the position and posture based on the bird's-eye view according to claim 1, characterized in that: The bird's-eye view image is a point cloud image after coordinate transformation, and the size parameters of the bird's-eye view image are the same as the size parameters of the original image.

3. The method for determining the position and posture based on the bird's-eye view according to claim 1, characterized in that: The first feature is a feature that is not affected by viewing angle changes.

4. The method for determining the position and posture based on a bird's-eye view according to claim 1, characterized in that: The deep learning network is a supervised deep learning network or an unsupervised deep learning network.

5. The method for determining the position and posture based on a bird's-eye view according to any one of claims 1 to 4, characterized in that: The performing mask processing on the moving target in the bird's-eye view to obtain the masked bird's-eye view comprises: Processing the bird's-eye view image using a target extraction network to obtain a bird's-eye view image with target detection results; Masking is performed on the detection target in the bird's-eye view with the target detection result to obtain a masked bird's-eye view.

6. The method for determining the position and posture based on a bird's-eye view according to claim 1, characterized in that: The fusing the first feature and the second feature to obtain the scene pose comprises: Convolving the first feature to obtain a first convolution feature; Convolving the second feature to obtain a second convolution feature; The first convolutional feature and the second convolutional feature are fused to obtain a scene pose.

7. The method for determining the position and posture based on the bird's-eye view according to claim 6, characterized in that: After fusing the first convolution feature and the second convolution feature to obtain the scene pose, the pose determination method based on the bird's-eye view further includes: When the dimension of the scene posture is different from the preset dimension, the dimension of the scene posture is adjusted so that the scene posture has the preset dimension.

8. A device for determining a posture based on a bird's-eye view, characterized in that: It includes a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement the method for determining a posture based on a bird's-eye view as described in any one of claims 1 to 7.

9. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed, the method for determining the posture based on the bird's-eye view described in any one of claims 1 to 7 is implemented.

Images