An autonomous mobile device and a control method thereof

Abstract

The application is suitable for the field of automatic driving perception technology, and provides an autonomous driving device and a control method thereof.The method comprises: continuously collecting images during driving of the autonomous driving device; completing one forward propagation by using a neural network model according to a first image collected at a first time T to synchronously output a plurality of task prediction results, wherein the neural network model comprises an image feature extraction module, a feature space conversion module, a time sequence information fusion module and a plurality of prediction modules; and adjusting a driving state of the autonomous driving device according to the plurality of task prediction results.Through the above method, during driving of the autonomous driving device, a unified feature is used to infer a plurality of task prediction results at the same time, so that each result of the inference has high consistency, the post-processing calculation amount is reduced, the task processing efficiency is significantly improved, and then the autonomous driving device can quickly and accurately respond to scene changes.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving perception technology, and in particular relates to an autonomous driving device and its control method. Background Technology

[0002] Occupancy perception technology originated from the occupancy grid mapping method in robotics. This method discretizes the environment into two-dimensional or three-dimensional grid cells, representing obstacle distribution through binary states (occupied / free), providing directly usable perception information for path planning. This method often involves using neural network models to process the initial collected data to obtain perception information. When using different models to handle multiple tasks, information transfer between different tasks can be disrupted, requiring not only complex post-processing fusion but also potentially resulting in conflicting results. Summary of the Invention

[0003] In view of this, this application provides an autonomous driving device and its control method, which can use unified features to infer multiple task prediction results simultaneously during the autonomous driving device's operation, making the inferred results highly consistent, reducing the post-processing computation, significantly improving task processing efficiency, and enabling the autonomous driving device to respond quickly and accurately to scene changes.

[0004] In a first aspect, this application provides an autonomous driving device, comprising: The vehicle body includes: a vehicle body; multiple drive wheels for moving the vehicle body; an image acquisition unit mounted on the vehicle body; a memory storing a neural network model, which includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules; and a processor configured to: control the autonomous driving device's movement; continuously acquire images using the image acquisition unit during the autonomous driving device's movement; at a first time T, perform a forward propagation based on the first image acquired at the first time T using the neural network model to simultaneously output multiple task prediction results; and based on the multiple task prediction results... As a result, the autonomous driving device adjusts its driving state; wherein, during the forward propagation process, the processor extracts two-dimensional image features from the first image through the image feature extraction module, projects the two-dimensional image features into three-dimensional space through the feature space conversion module to obtain three-dimensional features, the processor also receives a fusion request through the temporal information fusion module, fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features, and calculates temporal information based on the fused features, the processor also calculates the three-dimensional features and the temporal information through the multiple prediction modules respectively, so as to output the multiple task prediction results.

[0005] In some implementations, after projecting the two-dimensional image features onto the three-dimensional space, a dispersed feature point cloud is obtained. The processor then aggregates the feature point cloud to obtain the three-dimensional features, which are used to represent objects in the scene where the autonomous driving device is located.

[0006] In some implementations, the aggregation process is a pooling process.

[0007] In some embodiments, the processor is further configured to: maintain a queue through the time-series information fusion module, store the three-dimensional features output by the feature space conversion module each time into the queue, and when the queue reaches a preset length, remove the earliest three-dimensional feature stored in the queue from the queue; and record the three-dimensional features in the queue as historical three-dimensional features.

[0008] In some implementations, the processor extracts semantic information from the two-dimensional image features through the feature space transformation module. The semantic information includes target category and scene geometry. Based on the semantic information, the processor predicts the depth value of each pixel in the first image and projects the two-dimensional image features into the three-dimensional space based on the depth values.

[0009] In some implementations, the plurality of prediction modules include a semantic segmentation module, an instance segmentation module, and a velocity estimation module. The plurality of task prediction results include the geometric structure of the scene in which the autonomous driving device is located, the segmentation results of semantic information of different regions in the scene, the differentiation information of different individuals of the same target category in the scene, and the velocity of each voxel in the scene.

[0010] In some embodiments, the neural network model further includes an image feature enhancement module, and the processor is further configured to: process the two-dimensional image features through the image feature enhancement module to obtain two-dimensional image features at multiple scales, and input the two-dimensional image features at multiple scales to a feature space conversion module.

[0011] Secondly, this application provides a control method for an autonomous driving device, including: The autonomous driving device is controlled to drive; during the driving of the autonomous driving device, an image acquisition unit continuously acquires images; at a first time T, based on the first image acquired at the first time T, a forward propagation is completed using a neural network model to simultaneously output multiple task prediction results. The neural network model includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules; the driving state of the autonomous driving device is adjusted according to the multiple task prediction results; wherein, during the forward propagation process, the image feature extraction module extracts two-dimensional image features from the first image, the feature space transformation module projects the two-dimensional image features into three-dimensional space to obtain three-dimensional features, the temporal information fusion module receives a fusion request, fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features, and calculates temporal information based on the fused features; the multiple prediction modules respectively calculate the three-dimensional features and the temporal information to output the multiple task prediction results.

[0012] In some implementations, adjusting the driving state of the autonomous driving device based on the multiple task prediction results includes: constructing a map of the scene where the autonomous driving device is located based on the multiple task prediction results; and adjusting the driving state of the autonomous driving device based on the map.

[0013] In some implementations, the image feature extraction module, the feature space transformation module, the temporal information fusion module, and the multiple prediction modules interact through a standardized interface.

[0014] Thirdly, this application provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the control method provided in the second aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the control method provided in the second aspect above.

[0016] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the control method provided in the second aspect above.

[0017] The autonomous driving device provided in this application continuously acquires images using an image acquisition unit during its operation. Based on the first image acquired at a first time T, a neural network model is used to complete a forward propagation to simultaneously output multiple task prediction results. The neural network model includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules. The driving state of the autonomous driving device is adjusted based on the multiple task prediction results. Through this method, multiple task prediction results are simultaneously inferred using unified features during the autonomous driving device's operation, ensuring high consistency among the inferred results. Furthermore, completing multi-task prediction in a single forward propagation reduces computational load compared to a serial architecture. Additionally, projecting two-dimensional image features into three-dimensional space facilitates spatial understanding for multiple prediction modules and provides a foundation for subsequent temporal fusion. The temporal information fusion module combines the three-dimensional features extracted at the current time with three-dimensional features from past times (historical three-dimensional features), capturing dynamic changes in the scene and improving the coherence of perception and predictive capabilities.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the network structure of the neural network model provided in the embodiments of this application.

[0021] Figure 2This is a flowchart illustrating the control method for the autonomous driving device provided in the embodiments of this application.

[0022] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0027] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. This application provides an autonomous driving device, comprising: a vehicle body; multiple drive wheels for moving the vehicle body; an image acquisition unit disposed on the vehicle body; and a memory. The multiple drive wheels can be mounted on the vehicle body, and when the drive wheels rotate, they can move the vehicle body. The image acquisition unit can be a camera, such as a binocular camera or a trinocular camera. The camera can be mounted at the front of the vehicle body, and during the autonomous driving device's operation, it can continuously acquire images of the area in front of the device.

[0029] The memory stores a neural network model, which includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules. The processor is configured to: control the autonomous driving device's movement; continuously acquire images using the image acquisition unit during the autonomous driving device's movement; at a first time T, perform a forward propagation using the neural network model based on the first image acquired at the first time T, to simultaneously output multiple task prediction results; and adjust the driving state of the autonomous driving device based on the multiple task prediction results. During the forward propagation, the processor extracts two-dimensional image features from the first image using the image feature extraction module, projects the two-dimensional image features into three-dimensional space using the feature space transformation module to obtain three-dimensional features. The processor also receives a fusion request through the temporal information fusion module, fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features, and calculates temporal information based on the fused features. The processor also performs calculations on the three-dimensional features and the temporal information using the multiple prediction modules to output the multiple task prediction results.

[0030] Figure 1This is the structure of a neural network model, comprising an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules, each module being a part of the neural network model. In one forward propagation, the image is input from the image feature extraction module, processed by each module, and the prediction result is output from the prediction module. The autonomous driving device can be a car or a robot. During the autonomous driving device's operation, an image acquisition unit installed on the device can continuously acquire images; for example, the image acquisition unit can acquire images of the area in front of the autonomous driving device in real time at a preset frequency. Each image acquired by the image acquisition unit can be input to the image feature extraction module. The first time point T represents any moment during the autonomous driving device's operation. The image acquired by the image acquisition unit at the first time point T can be recorded as the first image. After acquiring the first image, it can be immediately input to the image feature extraction module to meet the real-time requirements of autonomous driving. The first image can be an RGB image. The image feature extraction module extracts two-dimensional image features from the first image, and the feature space transformation module projects these two-dimensional image features into three-dimensional space to obtain a large number of three-dimensional features. Two-dimensional image feature extraction extracts key information from the image for subsequent processing and analysis. For example, deep learning-based feature extraction methods can be used: Convolutional Neural Networks (CNNs) can automatically learn features in images. For instance, when using the AlexNet network structure for image feature extraction, after multiple layers of convolution, pooling, and fully connected operations, high-dimensional feature vectors can be extracted from the input image. If the feature space transformation module detects a moving object based on the three-dimensional features, it can initiate a fusion request to the temporal information fusion module. Based on the fusion request, the temporal information fusion module can fuse the currently received three-dimensional features and historical three-dimensional features to obtain fused features, and calculate temporal information based on the fused features. This temporal information helps the prediction module infer the motion of the moving object, such as its speed. The temporal information fusion module calculates temporal information only upon receiving a fusion request (e.g., upon detecting a moving object), rather than calculating it for every frame of data, thus reducing the average computational load. For example, the feature space transformation module only initiates a fusion request when a moving object is detected. The prediction module can be expanded according to actual needs; this embodiment does not limit the number of prediction modules. The task prediction results are used to represent relevant information about the scene in which the autonomous driving device is located, such as the type of objects in the scene and their speed. Based on multiple task prediction results, the processor can control the behavior of the autonomous driving device, thereby adjusting its driving state, such as deceleration, steering, or stopping. For example, if the processor determines that there is an animal in front of the autonomous driving device based on the task prediction results, it will control the autonomous driving device to stop moving.

[0031] In some embodiments, after projecting the two-dimensional image features onto the three-dimensional space, a dispersed feature point cloud is obtained. The processor then aggregates the feature point cloud to obtain the three-dimensional features, which represent objects in the scene where the autonomous driving device is located. Projecting two-dimensional image features onto three-dimensional space is a crucial step in achieving three-dimensional scene understanding from a two-dimensional image. For example, depth information can be used to map two-dimensional image features onto three-dimensional space. A feature point cloud is a set of discrete points obtained after projecting two-dimensional image features onto three-dimensional space. Each point contains three-dimensional coordinate information and attributes related to the original two-dimensional image features. The position of each feature point in three-dimensional space is represented by its x, y, and z coordinates. For example, in depth-based projection, the depth value directly determines the position of the feature point in the z-axis direction, while the image pixel coordinates determine the position in the x and y-axis directions. The aggregation processing of the feature point cloud is the process of integrating discrete feature point cloud data into three-dimensional features that can effectively represent objects in a three-dimensional scene. This process is crucial for subsequent scene understanding, object recognition, and decision-making by the autonomous driving device.

[0032] In some embodiments, the processor extracts semantic information from the two-dimensional image features through the feature space conversion module. The semantic information includes target category and scene geometry. Based on the semantic information, the processor predicts the depth value of each pixel in the first image and projects the two-dimensional image features into the three-dimensional space based on the depth values. Exemplarily, the feature space conversion module includes a monocular depth estimation network, whose main function is to convert two-dimensional image features into depth information, thereby providing a basis for subsequent 3D projection. This monocular depth estimation network adopts an encoder-decoder structure. The encoder layer abstracts the input two-dimensional image features layer by layer, extracting high-level semantic features. These features contain rich target category and structural information, such as human height proportions and vehicle size. The decoder layer, based on the semantic features extracted by the encoder, gradually recovers the image's depth information, generating a depth map corresponding to the input image size. The feature space conversion module also includes a feature projection layer, which uses the estimated depth information to project the two-dimensional image features into the three-dimensional space, obtaining a large number of dispersed feature point clouds.

[0033] In some embodiments, the aggregation process is a pooling process. When processing feature point clouds, pooling can aggregate scattered feature point clouds into a more compact 3D feature representation. For example, the feature space transformation module has a pooling layer that processes the feature point cloud according to a set aggregation rule (such as max pooling, average pooling, etc.).

[0034] In some embodiments, the processor is further configured to: maintain a queue through the time-series information fusion module, store the 3D features output by the feature space conversion module each time into the queue, and remove the earliest 3D feature stored in the queue when the queue reaches a preset length; and record the 3D features in the queue as historical 3D features. The main function of the time-series information fusion module is to manage historical 3D features so that this historical information can be used in subsequent processing. The queue stores the 3D features output by the feature space conversion module. By maintaining historical 3D features, the motion trajectory of objects can be tracked more accurately, and their future positions can be predicted. For example, a maximum queue length L is set, and each time the feature space conversion module outputs a new 3D feature, it is stored in the queue. When the queue length reaches L, the earliest stored 3D feature is removed, ensuring that the queue always stores the most recent and timely 3D features. Moreover, the dynamic adjustment mechanism of the queue avoids excessive memory consumption and improves the robustness of the system. When the feature space conversion module initiates a request to fuse historical feature information, the time-series information fusion module receives the current 3D features. The current 3D features are concatenated (fused) with historical 3D features in the queue. The concatenated features (fused features) are then fed into a fusion network for processing. The fusion network uses a convolutional neural network architecture to perform layer-by-layer computation on the input fused features, generating richer and more accurate temporal information for tasks such as object detection, semantic segmentation, and motion prediction. A queue is a linear data structure that follows the First-In-First-Out (FIFO) principle. It only allows elements to be inserted at one end (the tail) and deleted at the other end (the head). Basic queue operations include enqueueing, dequeueing, and viewing the head element (Front). For example, a queue can be implemented using an array.

[0035] In some embodiments, the temporal information fusion module may employ a convolutional neural network (CNN) or a long short-term memory network (LSTM).

[0036] In some embodiments, the plurality of prediction modules include a semantic segmentation module, an instance segmentation module, and a velocity estimation module. The prediction results of the plurality of tasks include the geometric structure of the scene in which the autonomous driving device is located, the segmentation results of semantic information of different regions in the scene, the differentiation information of different individuals of the same target category in the scene, and the velocity of each voxel in the scene. In this embodiment, the autonomous driving device focuses on three tasks: semantic segmentation, instance segmentation, and velocity estimation. Combining them can provide the autonomous driving device with a more comprehensive environmental perception capability.

[0037] In some embodiments, the semantic segmentation module may employ a Fully Convolutional Network (FCN), U-Net, DeepLab, or similar network. The instance segmentation module may employ a Mask R-CNN or YOLO network. The velocity estimation module may employ FlowNet or PWC-Net.

[0038] The semantic segmentation module performs geometric structure estimation and semantic information partitioning. Geometric structure estimation refers to estimating the geometric structure of the scene, which may involve predicting the scene's depth information, 3D shape, etc. For example, by predicting the depth value of each pixel or voxel, the 3D structure of the scene can be constructed. Semantic information partitioning refers to dividing the semantic information of different regions in the scene; in other words, it classifies each pixel or voxel in the scene into a predefined semantic category, such as roads, buildings, pedestrians, etc.

[0039] The instance segmentation module is responsible for distinguishing different individuals belonging to the same semantic category. For example, it identifies different vehicles, pedestrians, etc. in a scene and assigns a unique identifier to each individual.

[0040] The velocity estimation module voxels the scene, dividing the continuous scene space into discrete voxel units, each of which can contain a portion of one or more objects in the scene. The velocity of each voxel is then estimated, providing the autonomous vehicle with information about the motion of objects in the scene. This helps predict the future positions of objects, thus enabling safer navigation.

[0041] In some embodiments, the neural network model further includes an image feature enhancement module, and the processor is further configured to: process the two-dimensional image features through the image feature enhancement module to obtain two-dimensional image features at multiple scales, and input the two-dimensional image features at multiple scales to a feature space transformation module. Processing the two-dimensional image features through the image feature enhancement module can obtain image features at multiple scales. This is because features at different scales can capture information of different sizes and details in the image. For example, in object detection tasks, large-scale features are more suitable for detecting large objects, while small-scale features are more advantageous for detecting small objects. The feature space transformation module can fuse and integrate two-dimensional image features at multiple scales to generate a more representative and discriminative feature representation. For example, the image feature enhancement module can be implemented using the Spatial Pyramid Pooling (SPP) algorithm or the Feature Pyramid Network (FPN) algorithm.

[0042] During the operation of autonomous driving equipment, multiple task prediction results are inferred simultaneously using unified features, ensuring high consistency among the inferred results. Furthermore, completing multi-task predictions in a single forward propagation reduces computational load compared to a serial architecture. Additionally, projecting 2D image features into 3D space facilitates spatial understanding for multiple prediction modules and provides a foundation for subsequent temporal fusion. The temporal information fusion module combines the 3D features extracted at the current moment with 3D features from past moments (historical 3D features), capturing dynamic changes in the scene and improving the coherence of perception and predictive capabilities. Figure 2 A flowchart illustrating a control method for an autonomous driving device according to an embodiment of this application is shown below, and is described in detail below: Step 201: Control the autonomous driving device to drive.

[0043] Step 202: During the operation of the autonomous driving device, images are continuously acquired using the image acquisition unit.

[0044] Step 203: At the first time T, based on the first image acquired at the first time T, a forward propagation is completed using a neural network model to simultaneously output prediction results for multiple tasks.

[0045] Step 204: Adjust the driving status of the autonomous driving device based on the prediction results of the multiple tasks.

[0046] In this embodiment, the neural network model includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules. During the forward propagation process, the image feature extraction module extracts two-dimensional image features from the first image. The feature space transformation module projects the two-dimensional image features into a three-dimensional space to obtain three-dimensional features. The temporal information fusion module receives a fusion request and fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features. Temporal information is then calculated based on the fused features. The multiple prediction modules respectively calculate the three-dimensional features and the temporal information to output the prediction results for the multiple tasks. The control method provided in this embodiment can be specifically referred to in the relevant description of the above device embodiments, and will not be repeated here.

[0047] In some embodiments, adjusting the driving state of the autonomous driving device based on the prediction results of the multiple tasks includes: constructing a map of the scene where the autonomous driving device is located based on the prediction results of the multiple tasks; and adjusting the driving state of the autonomous driving device based on the map. The prediction results of the multiple tasks include semantic segmentation results, instance segmentation results, and velocity estimation results. Semantic segmentation results provide semantic information about different areas in the scene, such as roads, buildings, pedestrians, and vehicles. This information helps to build the basic framework of the map and clarify the function and type of different areas. Instance segmentation results can distinguish different individuals within the same semantic category, for example, marking different vehicles and pedestrians separately. This information can further refine the map, assigning a unique identifier to each individual, thereby achieving a precise representation of each object in the scene. Velocity estimation results provide velocity information for each voxel in the scene. This velocity information can be used to construct a dynamic map, helping to predict the future position of objects, thereby achieving more accurate real-time map updates. By combining these multi-task prediction results, a high-precision scene map containing both static and dynamic information can be constructed. Based on the constructed map, the autonomous driving device can plan the optimal path. Since the map is constructed based on dynamic information, the device can continuously adjust its driving state according to the real-time updated map information. For example, if a new obstacle is detected ahead, the device can adjust its path or speed in time to avoid a collision.

[0048] In some embodiments, the image feature extraction module, the feature space transformation module, the temporal information fusion module, and the multiple prediction modules interact through a standardized interface. This standardized interface allows each module (such as the image feature extraction module, feature space transformation module, temporal information fusion module, and multiple prediction modules) to be developed and tested independently. The standardized interface also provides good system scalability. When adding new functions or modules, it is only necessary to ensure that the new module conforms to the interface specification. Furthermore, maintaining and updating modules is more convenient because the coupling between modules is low.

[0049] As can be seen from the above, in this application's solution, multiple task prediction results are simultaneously inferred using unified features during the autonomous driving device's operation, ensuring high consistency among the inferred results. Furthermore, completing multi-task prediction in a single forward propagation reduces computational load compared to a serial architecture. Additionally, projecting two-dimensional image features into three-dimensional space facilitates spatial understanding for multiple prediction modules and provides a foundation for subsequent temporal fusion. The temporal information fusion module combines the three-dimensional features extracted at the current moment with those from past moments (historical three-dimensional features), capturing dynamic changes in the scene and enhancing the coherence of perception and predictive capabilities.

[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: at least one processor 30 ( Figure 3 (Only one is shown in the image), memory 31, and computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, it causes the electronic device to perform the steps of the control method embodiment described above.

[0051] The aforementioned electronic device 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0052] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0053] In some embodiments, the aforementioned memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. In other embodiments, the aforementioned memory 31 may be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the aforementioned memory 31 may include both internal storage units and external storage devices of the electronic device 3. The aforementioned memory 31 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 31 may also be used to temporarily store data that has been output or will be output. It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0054] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0055] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments described above.

[0056] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the steps described in the various method embodiments above.

[0057] This application provides a chip system including a processor, which is used to call and run a computer program from a memory, causing an electronic device equipped with the chip system to perform the steps in the various method embodiments described above.

[0058] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0059] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0060] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0061] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0062] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An autonomous driving device, characterized in that, include: Body; Multiple drive wheels, the drive wheels being used to move the vehicle body; An image acquisition unit is installed on the vehicle body; The memory stores a neural network model, which includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules. The processor is configured to: control the autonomous driving device to drive; continuously acquire images using the image acquisition unit during the driving of the autonomous driving device; at a first time T, perform a forward propagation using the neural network model based on the first image acquired at the first time T, so as to simultaneously output multiple task prediction results; and adjust the driving state of the autonomous driving device based on the multiple task prediction results. During the forward propagation process, the processor extracts two-dimensional image features from the first image through the image feature extraction module, projects the two-dimensional image features into three-dimensional space through the feature space conversion module to obtain three-dimensional features. The processor also receives a fusion request through the temporal information fusion module, fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features, and calculates temporal information based on the fused features. The processor also calculates the three-dimensional features and the temporal information through the multiple prediction modules to output the prediction results of the multiple tasks. If the feature space conversion module detects a moving object based on the three-dimensional features, it initiates the fusion request to the temporal information fusion module. The temporal information fusion module calculates the temporal information only when it receives the fusion request; it does not calculate the temporal information when it does not receive the fusion request.

2. The autonomous driving device as described in claim 1, characterized in that, After projecting the two-dimensional image features onto the three-dimensional space, a dispersed feature point cloud is obtained. The processor performs aggregation processing on the feature point cloud to obtain the three-dimensional features, which are used to represent objects in the scene where the autonomous driving device is located.

3. The autonomous driving device as described in claim 2, characterized in that, The aggregation process is a pooling process.

4. The autonomous driving device as described in claim 1, characterized in that, The processor is further configured to: maintain a queue through the time-series information fusion module, store the three-dimensional features output by the feature space conversion module each time into the queue, remove the earliest three-dimensional features stored in the queue from the queue when the queue reaches a preset length, and record the three-dimensional features in the queue as historical three-dimensional features.

5. The autonomous driving device as described in claim 1, characterized in that, The processor extracts semantic information from the two-dimensional image features through the feature space transformation module. The semantic information includes target category and scene geometry. Based on the semantic information, the processor predicts the depth value of each pixel in the first image and projects the two-dimensional image features into the three-dimensional space based on the depth value.

6. The autonomous driving device as described in claim 5, characterized in that, The multiple prediction modules include a semantic segmentation module, an instance segmentation module, and a velocity estimation module. The multiple task prediction results include the geometric structure of the scene where the autonomous driving device is located, the segmentation results of semantic information of different regions in the scene, the differentiation information of different individuals of the same target category in the scene, and the velocity of each voxel in the scene.

7. The autonomous driving device as described in claim 1, characterized in that, The neural network model further includes an image feature enhancement module, and the processor is further configured to: process the two-dimensional image features through the image feature enhancement module to obtain two-dimensional image features at multiple scales, and input the two-dimensional image features at multiple scales to the feature space conversion module.

8. A control method for an autonomous driving device, characterized in that, include: Control the autonomous driving device to drive; During the operation of the autonomous driving device, images are continuously acquired using an image acquisition unit; At the first time T, based on the first image acquired at the first time T, a forward propagation is completed using a neural network model to simultaneously output multiple task prediction results. The neural network model includes an image feature extraction module, a feature space transformation module, a temporal information fusion module, and multiple prediction modules. The driving status of the autonomous driving device is adjusted based on the prediction results of the multiple tasks. During the forward propagation process, the image feature extraction module extracts two-dimensional image features from the first image, and the feature space conversion module projects the two-dimensional image features into three-dimensional space to obtain three-dimensional features. The temporal information fusion module receives a fusion request, fuses the three-dimensional features with historical three-dimensional features according to the fusion request to obtain fused features, and calculates temporal information based on the fused features. The multiple prediction modules respectively calculate the three-dimensional features and the temporal information to output the prediction results of the multiple tasks. If the feature space conversion module detects a moving object based on the three-dimensional features, it initiates the fusion request to the temporal information fusion module. The temporal information fusion module calculates the temporal information only when it receives the fusion request; it does not calculate the temporal information when it does not receive the fusion request.

9. The control method as described in claim 8, characterized in that, The adjustment is based on the prediction results of the multiple tasks. The driving status of the autonomous driving device includes: Based on the prediction results of the multiple tasks, a map of the scene where the autonomous driving device is located is constructed; The autonomous driving device adjusts its driving status according to the map.

10. The control method as described in claim 8, characterized in that, The image feature extraction module, the feature space The conversion module, the time-series information fusion module, and the multiple prediction modules interact through a standardized interface.

Images