Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and relates to the technical field of data processing. According to the method, firstly, the feature map and the historical sampling point features of the image at the current moment are obtained, then, the current sampling point features meeting the set sampling rule are selected from the feature map, and finally, the current sampling point features and the historical sampling point features are fused to obtain the BEV features of the fused historical information at the current moment. The calculation amount and the access amount for transforming all the features in the historical feature map can be reduced, and the rate of fusing the historical BEV features can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] The information contained in a single frame image is limited, and during the driving process of the vehicle, some features of the current frame image may be lost due to reasons such as being blocked by objects.

[0003] For continuous surround camera images, the historical frame images of the surround camera contain feature information that is lost in the current frame image. Therefore, the feature information contained in the historical frame images can be fused with the feature information of the current frame image, that is, the feature information of the historical frame images is used to make up for the feature information lost in the current frame image, so as to improve the quality of the converted BEV features.

[0004] The existing fusion method first needs to transform the BEV feature map of the historical frame with the BEV feature map of the current frame according to the posture transformation of the vehicle motion information. Figure 1 First, align, then extract all the features of the BEV feature map, and finally fuse all the features of the current frame with all the features of the historical frames by splicing feature channels.

[0005] However, the above fusion method is used to combine the BEV feature maps of all historical frames with the BEV feature maps of the current frame. Figure 1 The image alignment process is complicated; and because all the features of the BEV feature map are usually selected for fusion during feature fusion, and all the features contain some unnecessary features. Therefore, the current method of using all the features of the BEV feature map for fusion processing will consume a lot of computer resources, causing a waste of computer resources and reducing the data processing rate. Summary of the invention

[0006] The present invention provides a data processing method, device and electronic device to improve the rate of feature fusion. The specific technical solution is as follows:

[0007] In a first aspect, the present application provides a data processing method, comprising:

[0008] Get the feature map of the current image and the features of historical sampling points;

[0009] From the feature map, select the current sampling point feature that meets the set sampling rule;

[0010] The current sampling point feature and the historical sampling point feature are fused to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusion of the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0011] Based on the above method, the amount of calculation and memory access for transforming all features in the historical feature graph can be reduced, and the rate of fusing historical BEV features can be improved.

[0012] In a possible implementation, obtaining the historical sampling point features includes:

[0013] In the BEV characteristic map at the current moment, creating a first reference point;

[0014] Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0015] Determine a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point;

[0016] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0017] By using the first and second reference points to perform feature sampling on the BEV feature map at the current moment and the historical BEV feature map respectively, and then fusing the first sampling point features and the second sampling point features obtained after sampling to obtain the historical sampling point features, the amount of calculation and memory access for transforming all features in the historical feature map can be reduced, computing resources can be saved, and the data processing rate can be improved.

[0018] In a possible implementation, the fusing the current sampling point feature and the historical sampling point feature to obtain the BEV feature fusing the historical information at the current moment includes:

[0019] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0020] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0021] The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

[0022] Based on the above method, the historical sampling point features and the current sampling point features can be fused according to the weights obtained after training the fully connected layer to obtain the BEV features that integrate the historical information.

[0023] In a possible implementation, selecting the current sampling point feature that satisfies a set sampling rule from the feature map includes:

[0024] In the feature map, creating a third reference point;

[0025] According to a set projection rule, the third reference point is projected to obtain a fourth reference point;

[0026] Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0027] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0028] Based on the above method, according to the third reference point created in the feature map of the current image, the current sampling point feature that meets the set sampling rule is selected, and the current sampling point feature is fused with the historical sampling point feature to obtain the BEV feature of the current moment that fuses the historical information. This can reduce the amount of calculation and memory access for transforming all features in the historical feature map and improve the rate of feature fusion.

[0029] In a second aspect, the present application provides a data processing device, comprising:

[0030] A data acquisition module is used to obtain the feature map of the current image and the features of historical sampling points;

[0031] A sampling module, used to select the current sampling point features that meet the set sampling rules from the feature map;

[0032] A data fusion module is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0033] In a possible implementation, the data acquisition module is specifically used to:

[0034] In the BEV characteristic map at the current moment, creating a first reference point;

[0035] Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0036] Determine a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point;

[0037] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0038] In a possible implementation, the data fusion module is specifically used to:

[0039] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0040] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0041] The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

[0042] In a possible implementation, the sampling module is specifically used to:

[0043] In the feature map, creating a third reference point;

[0044] According to a set projection rule, the third reference point is projected to obtain a fourth reference point;

[0045] Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0046] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0047] In a third aspect, the present application provides an electronic device, including:

[0048] Memory, used to store computer programs;

[0049] The processor is used to implement the steps of the above-mentioned data processing method when executing the computer program stored in the memory.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0051] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0053] Figure 2 A workflow diagram of the temporal attention model provided in the embodiment of the present application;

[0054] Figure 3 A timing tracking flow chart provided for an embodiment of the present application;

[0055] Figure 4 A simplified workflow diagram of the temporal attention model provided in the embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0059] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0060] For continuous surround camera images, the historical frame images of the surround camera contain feature information that the current frame image does not have or is missing. Therefore, the feature information contained in the historical frame images can be fused with the feature information of the current frame image, that is, the feature information of the historical frame images is used to make up for the feature information lost in the current frame image, so as to improve the quality of the converted BEV features.

[0061] The existing fusion method first needs to transform the BEV feature map of the historical frame with the BEV feature map of the current frame according to the posture transformation of the vehicle motion information. Figure 1 First, align, then extract all the features of the BEV feature map, and finally fuse all the features of the current frame with all the features of the historical frames by splicing feature channels.

[0062] However, the above fusion method is used to combine the BEV feature maps of all historical frames with the BEV feature maps of the current frame. Figure 1 The image alignment process is complicated; and because all the features of the BEV feature map are usually selected for fusion during feature fusion, and all the features contain some unnecessary features. Therefore, the current method of using all the features of the BEV feature map for fusion processing will consume a lot of computer resources, causing a waste of computer resources and reducing the data processing rate.

[0063] In view of this, in order to improve the rate of feature fusion, the present application provides a data processing method, which specifically includes: first obtaining the feature map of the image at the current moment and the historical sampling point features, then selecting the current sampling point features that meet the set sampling rules from the feature map, and finally fusing the current sampling point features and the historical sampling point features to obtain the BEV features of the current moment that fuse the historical information.

[0064] Through the method provided in the present application, the vehicle-side controller can select the current sampling point features that meet the set sampling rules from the feature map of the current image, and then fuse the current sampling point features with the historical sampling point features to obtain the BEV features of the current moment that fuse the historical information. This can reduce the amount of calculation and memory access required to transform all features in the historical feature map, thereby increasing the data processing rate.

[0065] Reference Figure 1 As shown, it is a flow chart of a data processing method provided in an embodiment of the present application, the method comprising:

[0066] S1, obtain the feature map of the current image and the features of the historical sampling points.

[0067] First of all, the method provided in this application can be applied to Figure 2 In the temporal attention model shown, the input of the temporal attention model can be continuous surround video frames, which can be obtained through image data collected by image sensors (such as any one or combination of surround view cameras, front view cameras, rear view cameras, and side view cameras), and the output of the temporal attention model can be BEV features that integrate historical information.

[0068] The temporal attention model includes: a flexible attention module and a cross attention module, wherein the flexible attention module is used to fuse historical features; the cross attention module is used to complete the extraction of BEV features.

[0069] In the embodiment of the present application, the temporal attention model can be physically deployed on a vehicle-side on-chip computing unit, such as a vehicle-side controller. The temporal attention model can also be trained on a cloud server platform. The present application does not impose any specific restrictions on the location of the deployment and training of the temporal attention model.

[0070] In the embodiment of the present application, the vehicle-side controller can sample the BEV feature map at the current moment and the historical BEV feature map, filter out the first sampling point feature from the BEV feature map at the current moment and filter out the second sampling point feature from the historical BEV feature map, and fuse the first sampling point feature and the second sampling point feature to obtain the historical sampling point feature. The specific process is as follows:

[0071] like Figure 2 As shown, the vehicle-side controller first obtains the BEV feature map at the current moment and the historical BEV feature map. The historical BEV feature map is the BEV feature map corresponding to the previous moment of the BEV feature map at the current moment. In the present application, the training Q value can be used to represent the BEV feature map at the current moment, and the previous frame Q value can be used to represent the historical BEV feature map.

[0072] Then, the BEV feature map at the current moment is initialized, that is, the training Q value is initialized; after the training Q value is initialized, the first reference point ( Figure 2 Then, according to the transformed posture of the historical BEV feature map and the BEV feature map at the current moment, the first reference point can be transformed into the historical BEV feature map, that is, the second reference point ( Figure 2 At this time, an index relationship between the first reference point at the current moment and the second reference point at the historical moment can be established. According to the index relationship, the position of the reference point at the previous moment corresponding to each current moment can be determined, wherein the transformation posture of the historical BEV characteristic map and the BEV characteristic map at the current moment can be determined by the vehicle's motion information, such as vehicle speed, deflection angle, etc. The present application does not impose any specific restrictions on the method of obtaining the transformation posture.

[0073] After determining the first reference point in the BEV feature map at the current moment and the second reference point corresponding to the first reference point in the historical BEV feature map, the vehicle-side controller can determine the first sampling coordinates of the first sampling point according to the first sampling offset between the first sampling point and the first reference point, that is, determine the coordinates of the first sampling point to be sampled in the BEV feature map at the current moment; and according to the second sampling offset between the second sampling point and the second reference point, the second sampling coordinates of the second sampling point can be determined, that is, determine the coordinates of the second sampling point to be sampled in the historical BEV feature map, wherein the sampling offset can be obtained by splicing the previous frame Q value and the training Q value to obtain the query Q value, using the query Q value as the query V value, and then training the query V value through a fully connected layer, and the previous frame Q value and the training Q value are spliced ​​to facilitate subsequent training and calculation; finally, according to the query Q value obtained by splicing the previous frame Q value and the training Q value, the query Q value is trained through the fully connected layer to obtain the weight, and the first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling point coordinate to obtain the historical sampling point feature.

[0074] In the embodiment of the present application, when the flexible attention module is used to fuse historical features, each fusion only needs to select some sampling point features in the BEV feature map of the previous frame for fusion. Since the BEV feature map of the previous frame contains the historical BEV feature information before the frame, the flexible attention module can establish time tracking between consecutive surround video frames. The flowchart of time tracking is shown in FIG. Figure 3 shown.

[0075] In a possible implementation, when the vehicle-side controller determines that the historical BEV feature map does not exist, that is, when the BEV feature map at the current moment is determined to be the first frame, the training Q value can be assigned to the Q value of the previous frame, that is, the BEV feature map at the current moment is determined to be the historical BEV feature map, and then the BEV feature map at the current moment and the historical BEV feature map are sampled, and the third sampling point feature is screened out from the BEV feature map at the current moment and the fourth sampling point feature is screened out from the historical BEV feature map, and the third sampling point feature and the fourth sampling point feature are fused, that is, self-attention is performed on the third sampling point feature in the BEV feature map at the current moment. The step of fusing the third sampling point feature in the BEV feature map at the current moment and the fourth sampling point feature in the historical BEV feature map can refer to the above-mentioned step of fusing the first sampling point feature of the BEV feature map at the current moment and the second sampling feature of the historical BEV feature map, which will not be repeated here.

[0076] Through the above method, according to the first reference point created in the BEV feature map at the current moment, a first sampling point feature whose distance offset with the first reference point is less than the set offset can be selected; according to the first reference point and the transformed posture of the BEV feature map at the current moment and the historical BEV feature map, the second reference point corresponding to the first reference point in the historical BEV feature map is determined, and the second sampling point feature whose distance offset with the second reference point is less than the set offset is selected; the first sampling point feature is fused with the second sampling point feature to obtain the historical sampling point feature that fuses the historical information; by using the first and second reference points to perform feature sampling on the BEV feature map at the current moment and the historical BEV feature map respectively, and fusing the first sampling point feature and the second sampling point feature obtained after sampling, the amount of calculation and memory access for transforming all the features in the historical feature map can be reduced, computing resources can be saved, and the data processing rate can be improved.

[0077] In an embodiment of the present application, after obtaining the historical sampling point features that integrate historical information, the vehicle-side controller can obtain a feature map of the image at the current moment. The feature map of the image at the current moment can be a multi-layer feature map and can be extracted through a feature extraction network. The present application does not impose any specific restrictions on the method of obtaining the feature map.

[0078] S2, select the current sampling point features that meet the set sampling rules from the feature map.

[0079] In the embodiment of the present application, after obtaining the feature map of the current moment image, the vehicle-side controller can first create a third reference point ( Figure 2voxel reference point in the image); then the third reference point is reprojected according to the camera parameters, and the third reference point is projected into the image coordinate system to obtain the fourth reference point ( Figure 2 ), and finally determine the position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point according to the coordinates of the fourth reference point and the above position deviation. In the present application, the sampling point coordinates can be obtained by summing the fourth reference point and the set convolution kernel offset, and the convolution kernel offset can be set according to actual application requirements.

[0080] After determining the coordinates of the sampling point to be selected in the image coordinate system, the vehicle-side controller can select the current sampling point features corresponding to the sampling point coordinates.

[0081] Through the above method, the current sampling point feature that meets the set sampling rule can be selected according to the third reference point created in the feature map of the image at the current moment.

[0082] S3, integrating the current sampling point features and the historical sampling point features to obtain the BEV features integrating the historical information at the current moment.

[0083] In an embodiment of the present application, after obtaining the historical sampling point features, the vehicle-side controller can learn the historical sampling point features through the fully connected layer to obtain the first weight (sampling point weight) corresponding to the historical sampling point features, and determine the second weight corresponding to the current sampling point features based on the first weight corresponding to the historical sampling point features; then, the historical sampling point features are weighted according to the first weight to obtain a first weighted result, and the current sampling point features are weighted according to the second weight to obtain a second weighted result; finally, the first weighted result and the second weighted result are summed to obtain the BEV features that integrate historical information at the current moment.

[0084] In the embodiment of the present application, in order to further improve the data processing rate, Figure 2In the BEV feature map at the current moment, a first reference point is created, and the reference point selection process of the second reference point corresponding to the first reference point in the historical BEV feature map is determined according to the transformed pose between the historical BEV feature map and the BEV feature map at the current moment. The process is converted into a reference point index table query process; a third reference point is created in the feature map of the image at the current moment, and the third reference point (voxel reference point) is projected according to the set projection rule (camera parameters) to obtain the fourth reference point (image reference point). The image reference point selection process is converted into an image coordinate mapping table query process. When the temporal attention model is trained in the cloud, the generation of reference points is related to the size of the transformed pose and the camera parameters, respectively. The size of the transformed pose is controllable, and the vehicle data of different models each corresponds to a reference point index table. The camera parameters are fixed for the same model. Therefore, by configuring the reference point index table and the image coordinate index table as offline, it can replace real-time repeated calculations, save the computing resources of the vehicle-side controller, and improve the data processing rate. The simplified temporal attention model is as follows Figure 4 shown.

[0085] In summary, the method provided by the present application samples the BEV feature map at the current moment and the historical BEV feature map by using the first and second reference points, respectively, and fuses the first sampling point features and the second sampling point features to obtain the historical sampling point features. It can reduce the amount of calculation and memory access for transforming all the features in the historical feature map, save computing resources, and improve the data processing rate; by learning the historical sampling point features, the weights corresponding to the historical sampling point features are obtained, and the current sampling point features in the current feature map are fused with the historical sampling point features according to the weights, so that the BEV features that fuse the historical information at the current moment can be extracted to make up for the feature information lost in the current feature map; and by configuring the reference point index table and the image coordinate index table as offline, it can replace the real-time repeated calculations, further save the computing resources of the vehicle-side controller, and improve the data processing rate.

[0086] Based on the method provided in the above embodiment, the embodiment of the present application also provides a data processing device, such as Figure 5 The figure is a schematic diagram of the structure of a data processing device in an embodiment of the present application, the device comprising:

[0087] The data acquisition module 501 is used to acquire the feature map of the current image and the features of the historical sampling points;

[0088] The sampling module 502 is used to select the current sampling point features that meet the set sampling rules from the feature map;

[0089] The data fusion module 503 is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment and the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

[0090] In a possible implementation, the data acquisition module 501 is specifically used for:

[0091] In the BEV characteristic map at the current moment, creating a first reference point;

[0092] Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment;

[0093] Determine a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point;

[0094] The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

[0095] In a possible implementation, the data fusion module 503 is specifically used for:

[0096] Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features;

[0097] Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result;

[0098] The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

[0099] In a possible implementation, the sampling module 502 is specifically configured to:

[0100] In the feature map, creating a third reference point;

[0101] According to a set projection rule, the third reference point is projected to obtain a fourth reference point;

[0102] Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation;

[0103] The current sampling point feature corresponding to the sampling point coordinates is selected.

[0104] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the functions of the aforementioned data processing device, referring to Figure 6 , the electronic device comprises:

[0105] At least one processor 601, and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 In the example, the processor 601 and the memory 602 are connected via a bus 600. The bus 600 is Figure 6 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and there is no limitation on the name.

[0106] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the data processing method discussed above by executing the instructions stored in the memory 602. The processor 601 can implement Figure 5 The functions of each module in the device shown.

[0107] Among them, the processor 601 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 602 and calling the data stored in the memory 602, the various functions of the device and processing data, the device can be monitored as a whole.

[0108] In one possible design, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0109] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the data processing method disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0110] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0111] By designing and programming the processor 601, the code corresponding to the data processing method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The steps of the data processing method of the embodiment shown are as follows: How to design and program the processor 601 is a technique well known to those skilled in the art and will not be described in detail here.

[0112] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the data processing method discussed above.

[0113] In some possible implementations, various aspects of the data processing method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the data processing method according to various exemplary implementations of the present application described above in this specification.

[0114] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, It is characterized in that include: Get the feature map of the current image and the features of historical sampling points; From the feature map, select the current sampling point feature that meets the set sampling rule; The current sampling point feature and the historical sampling point feature are fused to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusion of the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

2. The method according to claim 1, It is characterized in that The obtaining of historical sampling point features includes: In the BEV characteristic map at the current moment, creating a first reference point; Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment; Determine a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point; The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

3. The method according to claim 1 or 2, It is characterized in that The fusing the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fusing the historical information includes: Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features; Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result; The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

4. The method according to claim 1 or 2, It is characterized in that The step of selecting the current sampling point feature that satisfies the set sampling rule from the feature map includes: In the feature map, creating a third reference point; According to a set projection rule, the third reference point is projected to obtain a fourth reference point; Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation; The current sampling point feature corresponding to the sampling point coordinates is selected.

5. A data processing device, It is characterized in that include: A data acquisition module is used to obtain the feature map of the current image and the features of historical sampling points; A sampling module, used to select the current sampling point features that meet the set sampling rules from the feature map; A data fusion module is used to fuse the current sampling point feature and the historical sampling point feature to obtain the BEV feature of the current moment fused with the historical information, wherein the historical sampling point feature is obtained by fusing the first sampling point feature in the BEV feature map at the current moment and the second sampling point feature in the historical BEV feature map.

6. The device as claimed in claim 5, It is characterized in that The data acquisition module is specifically used for: In the BEV characteristic map at the current moment, creating a first reference point; Determine, based on the transformed posture of the historical BEV characteristic graph and the BEV characteristic graph at the current moment, a second reference point corresponding to the first reference point in the historical BEV characteristic graph, wherein the historical BEV characteristic graph is a BEV characteristic graph corresponding to a moment before the BEV characteristic graph at the current moment; Determine a first sampling coordinate of the first sampling point based on a first sampling offset between the first sampling point and the first reference point, and determine a second sampling coordinate of the second sampling point based on a second sampling offset between the second sampling point and the second reference point; The first sampling point feature corresponding to the first sampling coordinate is fused with the second sampling point feature corresponding to the second sampling coordinate to obtain the historical sampling point feature.

7. The device according to claim 5 or 6, It is characterized in that The data fusion module is specifically used for: Learning the historical sampling point features through a fully connected layer to determine a first weight corresponding to the historical sampling point features and a second weight corresponding to the current sampling point features; Weighting the historical sampling point features according to the first weight to obtain a first weighted result, and weighting the current sampling point features according to the second weight to obtain a second weighted result; The first weighted result and the second weighted result are summed to obtain the BEV feature integrated with the historical information at the current moment.

8. The device according to claim 5 or 6, It is characterized in that The sampling module is specifically used for: In the feature map, creating a third reference point; According to a set projection rule, the third reference point is projected to obtain a fourth reference point; Determine a position deviation between the sampling point to be selected and the fourth reference point, and determine the sampling point coordinates of the sampling point based on the coordinates of the fourth reference point and the position deviation; The current sampling point feature corresponding to the sampling point coordinates is selected.

9. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 4 when executing a computer program stored in the memory.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images