Data processing method and apparatus, and electronic device

Abstract

The present application relates to the technical field of data processing, and provides a data processing method and apparatus, and an electronic device. In the present application, firstly, a feature map of an image at a current moment and a historical sampling point feature are acquired, then a current sampling point feature meeting a set sampling rule is selected from the feature map, and finally, the current sampling point feature and the historical sampling point feature are fused to obtain a BEV feature of the current moment fused with historical information. By using such a method, the computational volume and the memory access cost of transforming all features in a historical feature map can be reduced, and the rate of fusing historical BEV features is improved.

Description

Data processing method, device and electronic equipment Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] The information contained in a single frame image is limited, and during the driving process of the vehicle, some features of the current frame image may be lost due to reasons such as being blocked by objects.

[0003] Summary of the Invention

[0004] This invention provides a data processing method, device, and electronic device to improve the rate of feature fusion. The specific technical solution is as follows:

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Obtain the feature map of the current image and the features of historical sampling points;

[0007] Selecting the current sampling point features that meet the set sampling rules from the feature map;

[0008] The current sampling point feature and the historical sampling point feature are fused to obtain a BEV feature at the current moment that is fused with historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0009] Based on the above method, the computational complexity and memory access required to transform all features in the historical feature graph can be reduced, thereby increasing the rate of fusing historical BEV features.

[0010] In a possible implementation, obtaining historical sampling point features includes:

[0011] Creating a first reference point in the BEV characteristic map at the current moment;

[0012] Determining, based on a transformed posture between the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0013] determining a first sampling coordinate of the first sampling point based on a first sampling offset between a first sampling point in the current BEV characteristic map and the first reference point, and determining a second sampling coordinate of the second sampling point based on a second sampling offset between a second sampling point in the historical BEV characteristic map and the second reference point;

[0014] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0015] By using the first and second reference points to sample the BEV feature map at the current moment and the historical BEV feature map respectively, and then fusing the first sampling point features and the second sampling point features obtained after sampling to obtain the historical sampling point features, the amount of calculation and memory access required to transform all features in the historical feature map can be reduced, computing resources can be saved, and the data processing rate can be improved.

[0016] In a possible implementation, the fusing the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with historical information includes:

[0017] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0018] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0019] The first weighted result and the second weighted result are summed to obtain the BEV feature of the current moment integrated with the historical information.

[0020] Based on the above method, the historical sampling point features and the current sampling point features can be fused according to the weights obtained after training the fully connected layer to obtain the BEV features that integrate historical information.

[0021] In a possible implementation, selecting the current sampling point feature that satisfies a set sampling rule from the feature map includes:

[0022] In the feature map, creating a third reference point;

[0023] Projecting the third reference point according to a set projection rule to obtain a fourth reference point;

[0024] Determining a position deviation between the sampling point to be selected and the fourth reference point, and determining the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0025] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0026] In one possible implementation, the method further includes:

[0027] In response to determining that the image at the current moment is the first frame, determining the BEV characteristic map at the current moment as a historical BEV characteristic map;

[0028] Sampling the BEV characteristic graph at the current moment and the historical BEV characteristic graph, filtering out a third sampling point feature from the BEV characteristic graph at the current moment and filtering out a fourth sampling point feature from the historical BEV characteristic graph;

[0029] The third sampling point feature and the fourth sampling point feature are fused to obtain the historical sampling point feature.

[0030] Based on the above method, according to the third reference point created in the feature map of the current moment image, the current sampling point feature that meets the set sampling rules is selected, and the current sampling point feature is fused with the historical sampling point feature to obtain the BEV feature of the current moment that integrates the historical information. This can reduce the computational complexity and memory access required to transform all features in the historical feature map, thereby improving the rate of feature fusion.

[0031] In a second aspect, the present application provides a data processing device, comprising:

[0032] Data acquisition module, used to obtain the feature map of the current image and the features of historical sampling points;

[0033] A sampling module, configured to select, from the feature map, features of a current sampling point that satisfy a set sampling rule;

[0034] A data fusion module is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0035] In a possible implementation, the data acquisition module is specifically configured to:

[0036] Creating a first reference point in the BEV characteristic map at the current moment;

[0037] Determining, based on a transformed posture between the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0038] determining a first sampling coordinate of the first sampling point based on a first sampling offset between a first sampling point in the current BEV characteristic map and the first reference point, and determining a second sampling coordinate of the second sampling point based on a second sampling offset between a second sampling point in the historical BEV characteristic map and the second reference point;

[0039] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0040] In a possible implementation, the data fusion module is specifically configured to:

[0041] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0042] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0043] The first weighted result and the second weighted result are summed to obtain the BEV feature of the current moment integrated with the historical information.

[0044] In a possible implementation, the sampling module is specifically configured to:

[0045] In the feature map, creating a third reference point;

[0046] Projecting the third reference point according to a set projection rule to obtain a fourth reference point;

[0047] Determining a position deviation between the sampling point to be selected and the fourth reference point, and determining the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0048] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0049] In a possible implementation, the data acquisition module is further configured to, in response to determining that the image at the current moment is the first frame, determine the BEV characteristic map at the current moment as a historical BEV characteristic map;

[0050] The sampling module is further configured to sample the current BEV characteristic graph and the historical BEV characteristic graph, and filter out a third sampling point feature from the current BEV characteristic graph and a fourth sampling point feature from the historical BEV characteristic graph;

[0051] The data fusion module is further configured to fuse the third sampling point feature and the fourth sampling point feature to obtain the historical sampling point feature.

[0052] In a third aspect, the present application provides an electronic device, comprising:

[0053] Memory for storing computer programs;

[0054] The processor is used to implement the steps of the above-mentioned data processing method when executing the computer program stored in the memory.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0056] For each of the above-mentioned aspects from the second to the fourth aspects and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG1 is a flow chart of a data processing method provided in an embodiment of the present application;

[0058] FIG2 is a flowchart of the workflow of the temporal attention model provided in an embodiment of the present application;

[0059] FIG3 is a timing tracking flow chart provided in an embodiment of the present application;

[0060] FIG4 is a flowchart of a simplified temporal attention model according to an embodiment of the present application;

[0061] FIG5 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0062] FIG6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail with reference to the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent the following two situations: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0064] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0065] For continuous surround-view camera images, the historical frames contain feature information that is absent or missing from the current frame. Therefore, the feature information contained in the historical frames can be fused with the feature information of the current frame. That is, the feature information of the historical frames is used to compensate for the missing feature information of the current frame, thereby improving the quality of the converted BEV (Bird's Eye View) feature.

[0066] The existing fusion method first needs to align the BEV feature map of the historical frame with the features in the BEV feature map of the current frame one by one according to the posture transformation of the vehicle motion information, then extract all the features of the BEV feature map of the historical frame and all the features of the BEV feature map of the current frame, and finally fuse all the features of the current frame with all the features of the historical frame by using feature channel splicing.

[0067] However, the image alignment process of aligning the BEV feature maps of all historical frames with the features of the BEV feature map of the current frame using the above fusion method is complex. Furthermore, since all features of the BEV feature map are typically selected for fusion during feature fusion, which includes some unnecessary features, the current method of fusing all features of the BEV feature map consumes a large amount of computer resources, resulting in a waste of computer resources and a reduction in data processing speed.

[0068] In view of this, in order to improve the rate of feature fusion, the present application provides a data processing method, including: first obtaining the feature map of the current moment image and the historical sampling point features, then selecting the current sampling point features that meet the set sampling rules from the feature map, and finally fusing the current sampling point features and the historical sampling point features to obtain the BEV features of the current moment fused with the historical information.

[0069] Through the method provided in this application, the vehicle-side controller can select the current sampling point features that meet the set sampling rules from the feature map of the current moment image, and then fuse the current sampling point features with the historical sampling point features to obtain the BEV features of the current moment that fuse the historical information. This can reduce the amount of calculation and memory access required to transform all features in the historical feature map, and improve the data processing rate.

[0070] 1 , which is a flow chart of a data processing method provided in an embodiment of the present application, the method includes steps S1 - S3 .

[0071] S1, obtain the feature map of the current image and the features of the historical sampling points.

[0072] First of all, the method provided in this application can be applied to the temporal attention model shown in Figure 2. The input of the temporal attention model can be continuous surround video frames. The continuous surround video frames can be obtained by image data collected by an image sensor (such as any one or combination of a surround camera, a front view camera, a rear view camera, and a side view camera). The output of the temporal attention model can be a BEV feature that integrates historical information.

[0073] The temporal attention model includes: a flexible attention module and a cross attention module, wherein the flexible attention module is used to fuse historical features; the cross attention module is used to complete the extraction of BEV features.

[0074] In the embodiments of this application, the temporal attention model can be physically deployed on an on-chip computing unit in the vehicle, such as a vehicle controller. Furthermore, the temporal attention model can be trained on a cloud server platform. This application does not impose any specific restrictions on the deployment or training location of the temporal attention model.

[0075] In the embodiment of the present application, the vehicle-side controller can sample the current BEV feature map and the historical BEV feature map, filter out the first sampling point feature from the current BEV feature map and the second sampling point feature from the historical BEV feature map, and fuse the first sampling point feature and the second sampling point feature to obtain the historical sampling point feature. The specific process is as follows:

[0076] As shown in Figure 2, the vehicle-side controller first obtains the current BEV feature map and the historical BEV feature map. The historical BEV feature map is the BEV feature map corresponding to the previous moment of the current BEV feature map. In this application, the training Q value can be used to represent the current BEV feature map, and the previous frame Q value can be used to represent the historical BEV feature map.

[0077] Then the BEV feature map at the current moment is initialized, that is, the training Q value is initialized; after the training Q value is initialized, a first reference point (i.e., the BEV reference point in Figure 2) can be created in the BEV feature map at the current moment according to the BEV grid information; then, according to the transformation posture of the historical BEV feature map and the BEV feature map at the current moment, the first reference point can be transformed into the historical BEV feature map, that is, the second reference point corresponding to the first reference point in the historical BEV feature map is determined (i.e., the historical frame transformation reference point in Figure 2). At this time, an index relationship between the first reference point at the current moment and the second reference point at the historical moment can be established. According to the index relationship, the position of the previous reference point corresponding to each current moment can be determined, wherein the transformation posture of the historical BEV feature map and the BEV feature map at the current moment can be determined by the motion information of the vehicle, for example, vehicle speed, deflection angle, etc. This application does not impose specific restrictions on the method of obtaining the transformation posture.

[0078] After determining the first reference point in the current BEV feature map and the second reference point corresponding to the first reference point in the historical BEV feature map, the vehicle-side controller can determine the first sampling coordinates of the first sampling point based on the first sampling offset between the first sampling point and the first reference point, that is, determine the coordinates of the first sampling point to be sampled in the current BEV feature map; and can determine the second sampling coordinates of the second sampling point based on the second sampling offset between the second sampling point and the second reference point, that is, determine the coordinates of the second sampling point to be sampled in the historical BEV feature map. The sampling offset can be obtained by concatenating the previous frame Q value and the training Q value to obtain a query Q value, using the query Q value as the query V value, and then training the query V value through a fully connected layer. The previous frame Q value and the training Q value are concatenated to facilitate subsequent training and calculation; finally, based on the query Q value obtained by concatenating the previous frame Q value and the training Q value, the query Q value is trained through a fully connected layer to fuse the first sampling point feature corresponding to the first sampling coordinate with the second sampling point feature corresponding to the second sampling point coordinate to obtain the historical sampling point feature.

[0079] In this embodiment of the present application, when using the flexible attention module to fuse historical features, each fusion only requires selecting a portion of the sampling point features in the BEV feature map of the previous frame for fusion. Because the BEV feature map of the previous frame contains historical BEV feature information before that frame, the flexible attention module can establish temporal tracking between consecutive surround view video frames. The flowchart of temporal tracking is shown in Figure 3.

[0080] In one possible implementation, when the vehicle-side controller determines that the historical BEV feature map does not exist, that is, when it determines that the BEV feature map at the current moment is the first frame, it can assign the training Q value to the Q value of the previous frame, that is, determine that the BEV feature map at the current moment is the historical BEV feature map. Then, the BEV feature map at the current moment and the historical BEV feature map are sampled, and the third sampling point feature is filtered out from the BEV feature map at the current moment, and the fourth sampling point feature is filtered out from the historical BEV feature map. The third sampling point feature and the fourth sampling point feature are fused, that is, self-attention is performed on the third sampling point feature in the BEV feature map at the current moment. The step of fusing the third sampling point feature in the BEV feature map at the current moment and the fourth sampling point feature in the historical BEV feature map can refer to the above-mentioned step of fusing the first sampling point feature of the BEV feature map at the current moment and the second sampling point feature of the historical BEV feature map, and will not be repeated here.

[0081] Through the above method, based on the first reference point created in the BEV feature map at the current moment, a first sampling point feature whose distance offset from the first reference point is less than the set offset can be selected; based on the transformed posture of the first reference point and the BEV feature map at the current moment and the historical BEV feature map, the second reference point corresponding to the first reference point in the historical BEV feature map is determined, and the second sampling point feature whose distance offset from the second reference point is less than the set offset is selected; the first sampling point feature is fused with the second sampling point feature to obtain the historical sampling point feature of the fused historical information; by using the first and second reference points to perform feature sampling on the BEV feature map at the current moment and the historical BEV feature map respectively, and fusing the first sampling point feature and the second sampling point feature obtained after sampling, the amount of calculation and memory access for transforming all features in the historical feature map can be reduced, computing resources can be saved, and the data processing rate can be improved.

[0082] In an embodiment of the present application, the feature map of the image at the current moment may be a multi-layer feature map and may be extracted through a feature extraction network. The present application does not impose any specific restrictions on the method for obtaining the feature map.

[0083] S2, select the current sampling point features that meet the set sampling rules from the feature map.

[0084] In an embodiment of the present application, after obtaining the feature map of the image at the current moment, the vehicle-side controller can first create a third reference point (i.e., the voxel reference point in FIG2 ) in the voxel coordinate system based on the BEV voxel grid information in a manner similar to the above-mentioned creation of the first reference point; then reproject the third reference point according to the camera parameters, project the third reference point to the image coordinate system, and obtain a fourth reference point (i.e., the image reference point in FIG2 ); finally, determine the position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the above-mentioned position deviation. In the present application, the sampling point coordinates can be obtained by summing the fourth reference point and the set convolution kernel offset, and the convolution kernel offset can be set according to the actual application requirements.

[0085] After determining the coordinates of the sampling point to be selected in the image coordinate system, the vehicle-side controller can select the current sampling point features corresponding to the sampling point coordinates.

[0086] In the above manner, the current sampling point feature that meets the set sampling rule can be selected based on the third reference point created in the feature map of the image at the current moment.

[0087] S3: Fuse the current sampling point features with the historical sampling point features to obtain the BEV features of the current moment that fuse the historical information.

[0088] In an embodiment of the present application, after obtaining the historical sampling point features, the vehicle-side controller can learn the historical sampling point features through the fully connected layer to obtain the first weight corresponding to the historical sampling point features (i.e., the sampling point weight in Figure 2), and determine the second weight corresponding to the current sampling point features based on the first weight corresponding to the historical sampling point features; then, the historical sampling point features are weighted according to the first weight to obtain a first weighted result; the current sampling point features are weighted according to the second weight to obtain a second weighted result; finally, the first weighted result and the second weighted result are summed to obtain the BEV features that integrate historical information at the current moment.

[0089] In an embodiment of the present application, in order to further improve the data processing rate, in FIG2, a first reference point is created in the BEV feature map at the current moment, and the reference point selection process of the second reference point corresponding to the first reference point in the historical BEV feature map is determined according to the transformed pose of the historical BEV feature map and the BEV feature map at the current moment is converted into a reference point index table query process; a third reference point is created in the feature map of the current moment image, and the third reference point (i.e., the voxel reference point in FIG2) is projected according to the set projection rule (e.g., camera parameters) to obtain the fourth reference point (i.e., the image reference point in FIG2). The image reference point selection process is converted into an image coordinate mapping table query process. When the temporal attention model is trained in the cloud, the generation of reference points is related to the size of the transformed pose and the camera parameters respectively. The size of the transformed pose is controllable. The vehicle data of different models each corresponds to a reference point index table, and the camera parameters are fixed for the same model. Therefore, by configuring the reference point index table and the image coordinate index table in an offline form, it is possible to replace real-time repeated calculations, save the computing resources of the vehicle-side controller, and improve the data processing rate. The simplified temporal attention model is shown in FIG4.

[0090] In summary, the method provided by the present application uses the first and second reference points to perform feature sampling on the current BEV feature map and the historical BEV feature map, respectively, and fuses the first sampling point features and the second sampling point features to obtain the historical sampling point features. This method can reduce the amount of computation and memory access required to transform all features in the historical feature map, save computing resources, and increase the data processing rate. By learning the historical sampling point features, the weights corresponding to the historical sampling point features are obtained. Based on the weights, the current sampling point features in the current feature map are fused with the historical sampling point features, and the BEV features that fuse the historical information at the current moment can be extracted to compensate for the feature information lost in the current feature map. Furthermore, by configuring the reference point index table and the image coordinate index table in an offline form, real-time repeated calculations can be replaced, further saving the computing resources of the vehicle-side controller and increasing the data processing rate.

[0091] Based on the method provided in the above embodiment, the embodiment of the present application further provides a data processing device. FIG5 is a schematic structural diagram of a data processing device in the embodiment of the present application, and the device includes:

[0092] Data acquisition module 501, used to obtain the feature map of the current image and the features of historical sampling points;

[0093] The sampling module 502 is used to select the current sampling point features that meet the set sampling rules from the feature map;

[0094] The data fusion module 503 is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0095] In a possible implementation, the data acquisition module 501 is specifically configured to:

[0096] Creating a first reference point in the BEV characteristic map at the current moment;

[0097] Determining, based on a transformed posture between the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0098] determining a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determining a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point;

[0099] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0100] In a possible implementation, the data fusion module 503 is specifically configured to:

[0101] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0102] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0103] The first weighted result and the second weighted result are summed to obtain the BEV feature of the current moment integrated with the historical information.

[0104] In a possible implementation, the sampling module 502 is specifically configured to:

[0105] In the feature map, creating a third reference point;

[0106] Projecting the third reference point according to a set projection rule to obtain a fourth reference point;

[0107] Determining a position deviation between the sampling point to be selected and the fourth reference point, and determining the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0108] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0109] Based on the same inventive concept, an electronic device is further provided in an embodiment of the present application. The electronic device can implement the functions of the aforementioned data processing device. Referring to FIG6 , the electronic device includes:

[0110] At least one processor 601, and a memory 602 connected to at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiments of the present application. Figure 6 takes the connection between the processor 601 and the memory 602 via the bus 600 as an example. The bus 600 is represented by a bold line in Figure 6, and the connection method between other components is only for schematic illustration and is not limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 6, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and there is no restriction on the name.

[0111] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can perform the data processing method discussed above by executing the instructions stored in the memory 602. The processor 601 can implement the functions of each module in the apparatus shown in Figure 5.

[0112] Among them, the processor 601 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 602 and calling data stored in the memory 602, the various functions of the device and processing data.

[0113] In one possible design, processor 601 may include one or more processing units. Processor 601 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0114] The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the data processing method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0115] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0116] By designing and programming the processor 601, the code corresponding to the data processing method described in the aforementioned embodiment can be embedded in the chip, so that the chip can execute the steps of the data processing method of the embodiment shown in FIG1 during operation. How to design and program the processor 601 is well known to those skilled in the art and will not be described in detail here.

[0117] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the data processing method discussed above.

[0118] In some possible implementations, various aspects of the data processing method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the data processing method according to various exemplary embodiments of the present application described above in this specification.

[0119] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0121] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0123] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data processing method, comprising: Get the feature map of the current image and the features of historical sampling points; From the feature map, select the current sampling point feature that meets the set sampling rule; The current sampling point feature and the historical sampling point feature are fused to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusion of the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

2. The method according to claim 1, characterized in that The obtaining of historical sampling point features includes: In the BEV characteristic map at the current moment, creating a first reference point; Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment; Determine a first sampling coordinate of the first sampling point based on a first sampling offset between a first sampling point in the BEV characteristic map at the current moment and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between a second sampling point in the historical BEV characteristic map and the second reference point; The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

3. The method according to claim 1 or 2, characterized in that The fusing the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fusing the historical information includes: Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features; Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result; The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

4. The method according to claim 1 or 2, characterized in that: The step of selecting the current sampling point feature that satisfies the set sampling rule from the feature map includes: In the feature map, creating a third reference point; According to a set projection rule, the third reference point is projected to obtain a fourth reference point; Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation; The current sampling point feature corresponding to the sampling point coordinates is selected.

5. The method of claim 1, further comprising: In response to determining that the current moment image is the first frame, determining the BEV characteristic map at the current moment as the historical BEV characteristic map; Sampling the BEV characteristic graph at the current moment and the historical BEV characteristic graph, filtering out a third sampling point feature from the BEV characteristic graph at the current moment and filtering out a fourth sampling point feature from the historical BEV characteristic graph; The third sampling point feature and the fourth sampling point feature are fused to obtain the historical sampling point feature.

6. A data processing device, comprising: A data acquisition module is used to obtain the feature map of the current image and the features of historical sampling points; A sampling module, used to select the current sampling point features that meet the set sampling rules from the feature map; A data fusion module is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

7. The device according to claim 6, characterized in that The data acquisition module is specifically used for: In the BEV characteristic map at the current moment, creating a first reference point; Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment; Determine a first sampling coordinate of the first sampling point based on a first sampling offset between a first sampling point in the BEV characteristic map at the current moment and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between a second sampling point in the historical BEV characteristic map and the second reference point; The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

8. The device according to claim 6 or 7, characterized in that The data fusion module is specifically used for: Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features; Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result; The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

9. The device according to claim 6 or 7, characterized in that The sampling module is specifically used for: In the feature map, creating a third reference point; According to a set projection rule, the third reference point is projected to obtain a fourth reference point; Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation; The current sampling point feature corresponding to the sampling point coordinates is selected.

10. The device according to claim 6, characterized in that The data acquisition module is further configured to determine the BEV characteristic diagram at the current moment as the historical BEV characteristic diagram in response to determining that the image at the current moment is the first frame; The sampling module is further used to sample the BEV characteristic graph at the current moment and the historical BEV characteristic graph, and filter out a third sampling point feature from the BEV characteristic graph at the current moment and filter out a fourth sampling point feature from the historical BEV characteristic graph; The data fusion module is further used to fuse the third sampling point feature and the fourth sampling point feature to obtain the historical sampling point feature.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing a computer program stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images