Model training method, electronic device, vehicle, medium, and program product

Abstract

The application relates to the technical field of vehicle driving, in particular to a model training method, an electronic device, a vehicle, a medium and a program product. The method comprises the following steps: obtaining training sample data, wherein the training sample data comprises a sample image, and the sample image comprises a target obstacle; in a label feature space corresponding to the sample image, the region corresponding to the target obstacle is expanded to obtain an adjusted label feature space; the sample image is input into a perception model to be trained to obtain a perception feature space of the sample image; and the model parameters of the perception model are adjusted based on the difference between the perception feature space and the adjusted label feature space. Based on this, the perception ability of the perception model can be improved, so that when the perception model is used for obstacle prediction subsequently, the vehicle can avoid missing identification or misidentifying the target obstacle.

Description

Technical Field

[0001] This application relates to the field of vehicle driving technology, and in particular to a model training method, electronic equipment, vehicle, medium, and program product. Background Technology

[0002] Electronic devices on a vehicle (such as the vehicle's infotainment system) can use perception models to predict obstacles such as road signs, warning signs, and lane curbs, and make decisions based on the perception results. For example, when the vehicle identifies a warning sign ahead based on the perception model, it may slow down.

[0003] However, in purely visual scenarios, there is a lack of specific distance measurement information for obstacles. This leads to a large error in the depth information of distant obstacles estimated by the perception model based on image data. Consequently, the perception error of distant obstacles in the vehicle's perception feature space is large, such as the inability to perceive distant obstacles, resulting in missed or misidentified distant obstacles. Summary of the Invention

[0004] This application provides a model training method, electronic device, vehicle, medium, and program product to improve the perception capability of the perception model and avoid the vehicle from missing or misidentifying distant obstacles.

[0005] Firstly, a model training method is provided for use in electronic devices. The method includes: acquiring training sample data, wherein the training sample data includes sample images, and the sample images include target obstacles; expanding the region corresponding to the target obstacle in the labeled feature space corresponding to the sample images to obtain an adjusted labeled feature space; inputting the sample images into a perception model to be trained to obtain the perception feature space of the sample images; and adjusting the model parameters of the perception model based on the difference between the perception feature space and the adjusted labeled feature space.

[0006] Understandably, during the training of the perception model, the region corresponding to the target obstacle in the labeled feature space is expanded to obtain an adjusted labeled feature space. In this adjusted space, the labeled region for the target obstacle is enlarged. This makes it easier for the spatial position of the target obstacle in the perception feature space to match the spatial position of the corresponding obstacle in the labeled feature space. This allows the perception model to learn the correct obstacle features from the region corresponding to the target obstacle in the adjusted labeled feature space, improving the model's perception capability, such as its recall ability when perceiving the target obstacle.

[0007] Correspondingly, when a vehicle uses the trained perception model to make predictions based on the collected image data, since the trained perception model has learned the correct features, when there is a large error in the depth information of distant obstacles estimated by the perception model based on the image data, the perception error of distant obstacles in the perception feature space can be reduced, thereby minimizing the possibility of the vehicle missing or misidentifying distant obstacles based on the perception model.

[0008] In one possible implementation of the first aspect described above, the labeled feature space and the perceptual feature space corresponding to the sample image are voxel spaces, and the region corresponding to the target obstacle is the region occupied by the labeled voxel set of the target obstacle.

[0009] In one possible implementation of the first aspect described above, the labeled voxel set includes M labeled voxels, where M is a positive integer greater than or equal to 1; in the labeled feature space corresponding to the sample image, the region corresponding to the target obstacle is expanded to obtain an adjusted labeled feature space, including: in the voxels of the labeled feature space, determining the extended voxel of each of the M labeled voxels; labeling the unlabeled extended voxels in the extended voxels of the i-th labeled voxel to obtain the adjusted labeled feature space, where i is a positive integer taking values ​​sequentially from 1 to M; wherein, the extended voxel of the i-th labeled voxel is: a voxel in the labeled feature space whose distance from the i-th labeled voxel is less than the i-th first distance and located on one side of the i-th labeled voxel along the i-th expansion direction.

[0010] In one possible implementation of the first aspect above, labeling the unlabeled extended voxels in the extended voxels of the i-th labeled voxel includes: changing the attributes of the unlabeled extended voxels in the extended voxels of the i-th labeled voxel to the same attributes as the i-th labeled voxel, wherein the attributes include: voxel occupancy status and obstacle category of the occupying voxel.

[0011] In one possible implementation of the first aspect described above, the electronic device is an electronic device on a vehicle. The position of the vehicle projected onto the labeled feature space is the first position; the i-th first distance is the distance between the first position and the position of the i-th labeled voxel, reduced by a first coefficient, and the i-th expansion direction is the direction from the first position to the position of the i-th labeled voxel; or; the i-th first distance is the distance between the position of the first position projected onto the horizontal plane and the position of the i-th labeled voxel projected onto the horizontal plane, reduced by a second coefficient, and the i-th expansion direction is the direction from the position of the first position projected onto the horizontal plane to the position of the i-th labeled voxel projected onto the horizontal plane.

[0012] In one possible implementation of the first aspect described above, the model parameters of the perception model are adjusted based on the difference between the perceptual feature space and the adjusted labeled feature space, including: calculating the deviation between the predicted occupancy probability of voxels in the perceptual feature space and the occupancy probability of voxels in the adjusted labeled feature space; and adjusting the model parameters of the perception model according to the deviation.

[0013] In one possible implementation of the first aspect mentioned above, the perception model includes an occupancy grid occupancy prediction model.

[0014] In a second aspect, an electronic device is provided, comprising: a memory for storing instructions; and at least one processor for executing the instructions to cause the electronic device to implement the first aspect described above and any of the various possible implementations of the first aspect described above, including a model training method.

[0015] Thirdly, a vehicle is provided, which stores instructions and executes the instructions on the vehicle, enabling the vehicle to implement any of the model training methods described in the first aspect and various possible implementations of the first aspect.

[0016] Fourthly, a computer-readable storage medium is provided, on which instructions are stored, which, when executed on a computer, cause the computer to execute any one of the model training methods described in the first aspect and various possible implementations of the first aspect.

[0017] Fifthly, a computer program product is provided, which, when run on a device, enables the device to implement the first aspect described above and any one of the various possible implementations of the first aspect described above as a model training method. Attached Figure Description

[0018] Figure 1 According to an embodiment of this application, a voxel space schematic diagram is shown;

[0019] Figure 2 According to some embodiments of this application, a planar schematic diagram of a labeled feature space and a perceived feature space is shown under a height H1 plane in a voxel space.

[0020] Figure 3 According to an embodiment of this application, a schematic diagram of a driving scenario for a vehicle 100 is shown;

[0021] Figure 4 According to some embodiments of this application, a schematic flowchart of a model training method is shown;

[0022] Figure 5 According to an embodiment of this application, a schematic diagram of the height H1 plane in another adjusted annotation feature space is shown;

[0023] Figure 6 According to an embodiment of this application, a two-dimensional coordinate diagram is shown;

[0024] Figure 7 According to some embodiments of this application, a schematic flowchart of a prediction method is shown;

[0025] Figure 8 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 200 is shown;

[0026] Figure 9 According to some embodiments of this application, a structural schematic diagram of a vehicle 100 is shown. Detailed Implementation

[0027] The illustrative embodiments of this application include, but are not limited to, model training methods, electronic devices, vehicles, media, and program products.

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] As described in the background section, in a purely visual scenario, there is a lack of specific distance measurement information for obstacles. This results in a large error in the depth information of distant obstacles estimated by the perception model based on image data. Consequently, the perception error of distant obstacles in the vehicle's perception feature space is large, leading to the vehicle missing or misidentifying distant obstacles.

[0030] The following section will explain in detail the principle behind vehicles missing or misidentifying distant obstacles by combining the training process of the perception model.

[0031] In some embodiments, sample images collected by the vehicle can be acquired to obtain training sample data. Furthermore, obstacles in the sample images are labeled beforehand to obtain a labeled feature space corresponding to the sample images. The training sample data is input into the perception model to be trained. The perception model performs depth estimation based on the sample images from the training sample data, and predicts the perception feature space corresponding to the image features based on the estimated depth information. This perception feature space includes the perception results of various obstacles in space. Based on the difference between the perception feature space and the labeled feature space, the model parameters of the perception model are adjusted to train the perception model. At this point, the vehicle can use the trained perception model to predict obstacle recognition. Understandably, the process of the perception model predicting the perception feature results of obstacles is the same as the method of obtaining the perception feature space based on sample images.

[0032] However, during the training process, when the perception model performs depth estimation based on sample images, the estimation of depth information for distant obstacles has a significant bias. This results in a large discrepancy between the perceived result of distant obstacles in the perception feature space and the corresponding obstacle's label in the labeled feature space. For example, the spatial position of a distant obstacle in the perceived result may differ significantly from the spatial position of the corresponding obstacle in the labeled feature space. Consequently, the perception model cannot correctly learn the features of the corresponding obstacle in the labeled feature space, leading to poor perception capabilities of the trained perception model. Therefore, when a vehicle uses the trained perception model to perceive obstacles in the collected image data, the depth information of distant obstacles estimated based on the image data will have a large error. The perception model, based on this inaccurate depth information, cannot obtain a correct perception result for distant obstacles, leading to the vehicle either missing or misidentifying distant obstacles.

[0033] For example, taking the labeled feature space and the perceived feature space as a voxel space, the perceived model includes an occupancy grid prediction model to predict the voxel occupancy status of each obstacle in the voxel space. The voxel space is a three-dimensional region divided into multiple cubes of a preset size, with each cube representing one voxel. Each voxel can be configured with attribute information, such as voxel occupancy status (including occupancy probability) and the semantic category of the occupier (e.g., car, person, flower bed, etc.). Figure 1 A voxel space is shown, in which each voxel, for example, the first voxel T1, can have corresponding attribute information.

[0034] For ease of explanation, Figure 2A schematic diagram of the labeled feature space and the perceptual feature space is shown below a voxel space at height H1. Each square in the diagram represents a voxel at the same height. The labeled voxels for obstacle 1 in the labeled feature space include labeled voxels A1, A2, and A3. The perceptual voxels for obstacle 1 in the perceptual feature space include perceptual voxels B1, B2, and B3. Comparing the labeled feature space and the perceptual feature space, the positions of labeled voxels A1, A2, and A3 differ significantly from those of perceptual voxels B1, B2, and B3. Therefore, the semantic category of the attribute in the corresponding position of perceptual voxels B1, B2, and B3 in the labeled feature space is empty, and the voxel occupancy state is unoccupied (i.e., occupancy probability is 0). At this point, the perception model learns incorrect features based on the voxels at corresponding locations in the labeled feature space, such as believing there are no obstacles there. When the vehicle uses the trained perception model for prediction, the depth information of distant obstacles estimated based on image data will have a large error, resulting in a large perception error for distant obstacles in the vehicle's obtained perception feature space. For example, it may perceive obstacle 1 as occupying perception voxels B1, B2, and B3, which could lead to missed or misidentified distant obstacles.

[0035] Figure 3 A schematic diagram of a driving scenario for a vehicle 100 is shown according to an embodiment of this application. As shown in the figure, obstacle 1 is used as... Figure 3 Take fault sign 01 as an example. If vehicle 100 fails to recognize fault sign 01 in the distance ahead, vehicle 100 will be unable to issue a correct warning.

[0036] Therefore, to solve the above problems, this application proposes a model training method. In this method, training sample data is obtained, which includes sample images, and the sample images include target obstacles. In the labeled feature space corresponding to the sample images, the region corresponding to the target obstacles is enlarged to obtain an adjusted labeled feature space. The sample images are input into the perception model to be trained to obtain the perception feature space of the sample images. Based on the difference between the perception feature space and the adjusted labeled feature space, the model parameters of the perception model are adjusted.

[0037] For example, consider a voxel space as both the perceptual feature space and the labeled feature space. The perception result of the target obstacle M1 in the perceptual feature space includes the second voxel T2. However, by expanding the region corresponding to the target obstacle M1 in the labeled feature space, the voxel corresponding to the second voxel T2 in the adjusted labeled feature space is occupied by the target obstacle M1. Thus, the perceptual model can learn the features of the target obstacle M1, thereby avoiding the inability to perceive the target obstacle M1.

[0038] Understandably, during the training of the perception model, the region corresponding to the target obstacle in the labeled feature space is expanded to obtain an adjusted labeled feature space. The labeled region for the target obstacle in this adjusted feature space is larger. The spatial position of the target obstacle in the perception feature space is more easily matched with the corresponding spatial position of the obstacle in the adjusted labeled feature space. This allows the perception model to learn the correct obstacle features from the region corresponding to the target obstacle in the adjusted labeled feature space, improving the perception model's perception capability, such as its recall ability when perceiving the target obstacle. Correspondingly, when the vehicle uses the trained perception model to make predictions based on the collected image data, because the trained perception model has learned the correct features, when there is a large error in the depth information of distant obstacles estimated by the perception model based on the image data, the perception error for distant obstacles in the perception feature space can be reduced, thereby minimizing the possibility of the vehicle missing or misidentifying distant obstacles based on the perception model.

[0039] In some embodiments, the target obstacle may include, but is not limited to, static obstacles such as speed limit signs, flower beds, curbs, and warning signs. The target obstacle may be a distant obstacle.

[0040] Understandably, obstacle ranging accuracy is poor at long distances. Therefore, the area corresponding to the marked voxels of distant obstacles can be enlarged only, while the area of ​​nearby obstacles is not enlarged, thus ensuring the perception accuracy at close range.

[0041] Figure 4 According to some embodiments of this application, a schematic flowchart of a model training method is shown. An electronic device is used as the execution entity. It is understood that the electronic device in the embodiments of this application can include, but is not limited to, any device that can be used for model data processing, including but not limited to mobile phones, in-vehicle systems, terminals in self-driving cars, computers, servers, etc. In some embodiments, the electronic device can be an electronic device on a vehicle. The specific steps are as follows:

[0042] S401, Obtain training sample data, wherein the training sample data includes sample images, and the sample images include target obstacles.

[0043] In some embodiments, the training sample data includes sample images collected by each vehicle, which may be collected by the same vehicle or by different vehicles.

[0044] S402, in the labeled feature space corresponding to the sample image, the region corresponding to the target obstacle is expanded to obtain the adjusted labeled feature space.

[0045] In some embodiments, after acquiring a sample image, the positions of each obstacle in the sample image are labeled in voxel space to create a labeled feature space. This labeled feature space contains the voxels occupied by each obstacle. For example, the positions of obstacles in the sample image within a 3D space centered on the vehicle acquiring the sample image (hereinafter referred to as "the vehicle") are obtained using radar point clouds. The 3D space centered on the vehicle is then divided into voxels according to a preset size, and the voxels corresponding to the obstacles are labeled to obtain the labeled feature space. The region corresponding to the target obstacle in the labeled feature space corresponding to the sample image is then enlarged to obtain an adjusted labeled feature space.

[0046] The following section details how the adjusted labeled feature space is obtained.

[0047] In some embodiments, the labeled feature space and the perceptual feature space corresponding to the sample image can be voxel spaces, and the region corresponding to the target obstacle is the region occupied by the labeled voxel set of the target obstacle. The region occupied by the labeled voxel set of the target obstacle is expanded to obtain the adjusted labeled feature space.

[0048] The process of expanding the area occupied by the labeled voxels of the target obstacle to obtain an adjusted labeled feature space can include: determining the extended voxels of each of the M labeled voxels in the labeled feature space; labeling the unlabeled extended voxels in the extended voxels of the i-th labeled voxel to obtain the adjusted labeled feature space, where i is a positive integer taking values ​​sequentially from 1 to M; wherein the extended voxels of the i-th labeled voxel can be: voxels in the labeled feature space whose distance from the i-th labeled voxel is less than the i-th first distance and located on one side of the i-th labeled voxel along the i-th expansion direction. Then, changing the attributes of the unlabeled extended voxels in the extended voxels of the i-th labeled voxel to the same attributes as the i-th labeled voxel, wherein the attributes include: voxel occupancy status and obstacle category of the occupying voxel.

[0049] For example, Figure 5A schematic diagram of the height H1 plane in the adjusted annotation feature space is shown. The diagram shows the extended voxels: A1 (less than the first first distance D11 in direction F1), A2 (less than the second first distance D22 in direction F2), and A3 (less than the third first distance D33 in direction F3). Furthermore, voxels A1, A2, and A3 annotate the unannotated voxels in the extended voxels, assuming that the extended voxels also correspond to obstacle 1. In the adjusted annotation feature space, the semantic category of the attributes of the corresponding voxels of perceptual voxels B1, B2, and B3 is obstacle 1, and their occupancy status is occupied (i.e., occupancy probability is 1). The perception model learns the features of obstacle 1 based on the voxels at corresponding locations in the adjusted labeled feature space. This allows the perception model to subsequently perceive obstacle 1, preventing the vehicle from missing or misidentifying distant obstacle 1 using the trained perception model. Understandably, due to the small size of voxels, the distance between voxels can be either the distance between voxel centers or the distance between any point within one voxel and any point within another voxel; the specific distance between voxels is not limited here.

[0050] In some implementations, the position of the vehicle projected onto the labeled feature space is called the first position. The i-th first distance can be the distance between the first position and the location of the i-th labeled voxel, reduced by a first coefficient. The i-th expansion direction can be the direction from the first position to the location of the i-th labeled voxel. For example, the first coefficient can be 0.1. Assuming the first position is the origin in the labeled feature space, and the distance between the first position and the location of the i-th labeled voxel is D1, then the i-th first distance is 0.1*D1.

[0051] Understandably, the voxel in the i-th expansion direction and the first position corresponding to the vehicle can be approximately located on the same ray. Voxels in the i-th expansion direction on the same ray may project onto the same pixel in the image. The deviation of the spatial position of the target obstacle in the perception feature space predicted by the perception model based on image depth information is very likely to be a deviation of different positions on the same ray. When the labeled voxels of the target obstacle are expanded according to the corresponding expansion direction, even if there is a large deviation in the perception of the spatial position of the target obstacle in the perception feature space predicted by the perception model based on image depth information, the expanded voxels in the adjusted labeled feature space can still match. This allows the perception model to learn the correct obstacle features from the region corresponding to the target obstacle in the adjusted labeled feature space, improving the perception model's perception capability, such as its recall capability when perceiving the target obstacle. This avoids subsequent missed or incorrect predictions by the vehicle based on the perception model.

[0052] In other implementations, the i-th first distance can also be: the distance between the first position projected onto the horizontal plane and the i-th labeled voxel projected onto the horizontal plane, reduced by the second coefficient. The i-th expansion direction can also be: the direction from the first position projected onto the horizontal plane to the i-th labeled voxel projected onto the horizontal plane. For example, the second coefficient can be 0.1. Assuming the first position is the origin in the labeled feature space, and the distance between the first position projected onto the horizontal plane and the i-th labeled voxel projected onto the horizontal plane is D2, then the i-th first distance is 0.1*D2.

[0053] The following example illustrates how to determine the expanded voxel of the i-th annotation voxel among M annotation voxels, with the first distance being the distance between the first position projected onto the horizontal plane and the i-th annotation voxel projected onto the horizontal plane, reduced by the second coefficient, and the i-th expansion direction being the direction from the first position projected onto the horizontal plane to the i-th annotation voxel projected onto the horizontal plane.

[0054] Assume the i-th annotation voxel in the annotation feature space lies in a plane of height H1. The first position is the origin O in the annotation feature space. Projecting the origin O onto the plane of height H1 yields the first position, which is at the same height as the i-th annotation voxel. The three-dimensional coordinates of the first position and the i-th annotation voxel can be simplified to two-dimensional coordinates for calculation.

[0055] Assuming the two-dimensional coordinates of the first position on the height H1 plane are (0,0) and the coordinates of the i-th labeled voxel are (xi,yi), the ray parameter equation from (0,0) to (xi,yi) is calculated, and the following formula (1) is obtained.

[0056] Formula (1): y = yi / xi * x

[0057] Where x is greater than xi. (x, y) is any point on the ray that satisfies the condition that the distance to (xi, yi) is less than the i-th first distance. For example, refer to... Figure 6 As shown, di is the i-th first distance corresponding to the i-th labeled voxel. A line segment on the ray that satisfies a distance to (xi,yi) less than the i-th first distance is denoted as Li. The i-th first distance can be the product of the distance Si from (0,0) to (xi,yi) and the second coefficient. It is understandable that as Si increases, di increases. In line segment Li, sampling can be performed with a sampling step size of t1, and the voxel corresponding to the sampled position can be used as the extended voxel of the i-th labeled voxel. It is understandable that when the sampled position is a decimal, the extended voxel can be selected using interpolation, such as selecting the voxel closest to the sampled position as the extended voxel using nearest neighbor interpolation. Assuming i is 3, the obtained coordinates (x3, y3) are... Figure 5 The coordinates of the labeled voxel A3 are given. Furthermore, the sampled position of labeled voxel A3 is... Figure 5 The positions of points G1, G2, and G3 corresponding to the given information can be used to determine the location of the points G1, G2, and G3. Figure 5 The voxel A3 shown is an extended voxel in direction F1.

[0058] Understandably, by projecting voxels onto a horizontal plane and combining this with ray equations, the computational workload for determining extended voxels can be significantly reduced.

[0059] Once the extended voxels for each labeled voxel are determined, a set of extended voxels is obtained. These extended voxels are then labeled. For example, if an extended voxel is not labeled, the attribute of the extended voxel of the i-th labeled voxel is modified to be the same as that of the i-th labeled voxel. For instance, suppose the attribute of the i-th labeled voxel is "obstacle 1," and its state is "occupied" (i.e., the voxel occupancy probability is 1). If one of the extended voxels C1 of the i-th labeled voxel is unlabeled (i.e., the obstacle category in the attribute of extended voxel C1 is empty), and its state is "unoccupied" (voxel occupancy probability is 0), then the attribute of extended voxel C1 is modified to "obstacle category 1," and the voxel occupancy probability is 1. If the extended voxel has already been labeled, its original labeled attributes are retained. For instance, suppose the attribute of extended voxel C1 is "obstacle category 2," and the voxel occupancy probability is 1. In this case, the attribute of extended voxel C1 is not modified.

[0060] S403, input the sample image into the perceptual model to be trained to obtain the perceptual feature space of the sample image.

[0061] In some embodiments, after the perception model to be trained acquires a sample image, it first extracts 2D image features from the sample image, and then estimates the depth information of each pixel based on the image features. Subsequently, it combines the camera intrinsic and extrinsic parameters, projects the 2D image features with depth information into the voxel space, and rasterizes them to generate 3D voxel features. Based on the features in the voxel space, it determines whether each voxel is occupied by an obstacle and the type of obstacle, obtains the perception results of each obstacle, and thus forms a perception feature space containing spatial location and semantic information.

[0062] S404, based on the difference between the perceptual feature space and the adjusted labeled feature space, adjusts the model parameters of the perceptual model.

[0063] In some embodiments, the deviation between the predicted occupancy probability of voxels in the perceptual feature space and the occupancy probability of voxels in the adjusted labeled feature space is calculated; and the model parameters of the perceptual model are adjusted based on the deviation.

[0064] For example, the deviation between the predicted occupancy probability of voxels in the perceptual feature space being occupied by various obstacles and the adjusted occupancy probability of voxels in the labeled feature space being occupied by various obstacles is calculated using a loss function, and the model parameters of the perceptual model are iterated based on the deviation value obtained from the loss function.

[0065] The following formula (2) shows a loss function formula.

[0066] Formula (2)

[0067] Where a1 and a2 are preset weighting coefficients. Let be the loss function for the voxel occupancy state in the perceptual feature space. L1 is the loss function for the obstacle category occupied by each voxel in the perception feature space. The following formulas (3) and (4) show specific expressions for L1 and L2, respectively.

[0068] L1= Formula (3)

[0069] L2= Formula (4)

[0070] Where N is the number of voxels in the perceptual feature space. The occupancy probability of the i-th voxel is 0 or 1. Let be the predicted occupancy probability of the i-th voxel, with a value ranging from 0 to 1. M is the total number of voxels in the labeled feature space that are in the occupied state. If the occupancy probability of a voxel is 1 (as a mask), then the voxel's... =1, otherwise It is 0. The category label for the k-th class of the i-th voxel. Let be the predicted class probability of the i-th voxel in the k-th class.

[0071] Figure 7 According to some embodiments of this application, a schematic flowchart of a prediction method is shown, applied to electronic devices (e.g., in-vehicle infotainment systems) in vehicles. The specific process includes:

[0072] S701, acquire at least one image data based on vehicle data; the at least one image data includes at least one target obstacle.

[0073] In some embodiments, electronic devices on the vehicle can acquire multiple image data collected by the vehicle.

[0074] S702, based on at least one image data, a perception feature space is predicted using a perception model, the perception feature space including the perception results of the target obstacle.

[0075] Understandably, this perception model can be used to... Figure 4 The perceptual model obtained by the method shown.

[0076] In some embodiments, after the vehicle acquires image data, it inputs the currently acquired image data into a trained perception model to obtain a perception feature space. The perception result for the target obstacle includes the predicted occupancy probability of the target obstacle.

[0077] For example, the predicted occupancy probability of voxel K is 0.8, and if the voxel is occupied, the probability that it belongs to the first type of target obstacle is 0.8.

[0078] S703, based on the perception results of the target obstacle, obtain the identification result of the target obstacle.

[0079] For example, based on the preset confidence levels of predicted occupancy probability and category probability, and the perception results of the target obstacle, voxels are selected to obtain voxels that meet the confidence requirements. These voxels are then clustered through neighborhood connected components to obtain the corresponding regions of the target obstacle. Based on the corresponding regions of the target obstacle, core perception features, including real-world coordinates, size, dominant category, and overall confidence, are aggregated and calculated to output the recognition results of the target obstacle.

[0080] Furthermore, after acquiring the identification results of the target obstacle, the vehicle's electronic equipment (such as the in-vehicle infotainment system) can provide correct warning information based on the identification results of the target obstacle. The Adaptive Cruise Control (ACC) unit can also respond correctly based on the identification results of the target obstacle, thereby improving the user experience.

[0081] It is understood that in other embodiments, depending on actual needs, the steps shown in the above embodiments can be combined, deleted, or replaced with other steps that are beneficial to achieving the purpose of this application, and this application does not impose any restrictions here.

[0082] This application also provides a schematic diagram of the structure of an electronic device 200. The electronic device 200 may include at least one processor that executes instructions stored in a non-transitory computer-readable medium such as memory. The processor may be any conventional processor, such as a commercially available CPU. Optionally, the processor may be a special-purpose device such as an ASIC or other hardware-based processor. The memory may contain instructions (e.g., program logic) that can be executed by the processor to perform various functions of the vehicle 100, including the functions of the embodiments described above.

[0083] Next, combine Figure 8 The structure of electronic device 200 will be described.

[0084] like Figure 8 As shown, the electronic device 200 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic unit 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) device 105. Wherein:

[0085] Processor 101 may include one or more processing units, such as processing modules or circuits that include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, field programmable gate array (FPGA), neural network processing unit (NPU), etc., and may include one or more single-core or multi-core processors. In some embodiments, the CPU may be used to optimize the neural network model to be run, and the NPU may be used to run the neural network model to be run. Processor 101 is used to execute any of the above-described model training methods.

[0086] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 102 can be used to store data provided by the aforementioned different services, such as sensor data, image data, or video data, and can also be used to store instructions for the model training methods provided in the aforementioned embodiments.

[0087] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card. In other embodiments, the non-volatile memory 103 may be used to store instructions for the model training methods provided in the foregoing embodiments.

[0088] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 200 to implement the model training methods provided in the embodiments of this application.

[0089] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 200, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 200, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 200 may communicate with other devices through the communication interface 104, for example, the electronic device 200 may obtain relevant data from other devices through the communication interface 104.

[0090] Input / output (I / O) device 105 can be an input device such as a keyboard or mouse, and an output device such as a monitor. Users can interact with electronic device 200 through input / output (I / O) device 105.

[0091] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface to other modules of the electronic device 200. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface to the system memory 102 and the non-volatile memory 103.

[0092] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for the system control logic unit 106 to form a system-on-chip (SoC).

[0093] Understandable. Figure 8 The structure of the electronic device 200 shown is merely an example. In other embodiments, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0094] It is understood that the model training method and prediction method mentioned in the embodiments of this application can be applied to electronic devices in vehicle 100. Figure 9 This is a schematic diagram of a possible functional framework of a vehicle 100 provided in an embodiment of this application.

[0095] like Figure 9 As shown, the functional framework of vehicle 100 may include various subsystems, such as the sensor system 10, control system 20, one or more peripheral devices 30 (one is shown as an example), power supply 40, and computer system 50. Optionally, vehicle 100 may also include other functional systems, such as an engine system that provides power to vehicle 100, etc., which are not limited herein. The sensor system 10 may include several detection devices that can sense the measured information and convert the sensed information into electrical signals or other required forms of information output according to a certain rule.

[0096] As shown in the figure, these detection devices may include a Global Positioning System (GPS) 11, a vehicle speed sensor 12, an Inertial Measurement Unit (IMU) 13, etc., and this application is not limited thereto. The GPS 11 is a system that uses GPS positioning satellites to perform real-time positioning and navigation globally. In this application, the GPS 11 can be used to achieve real-time positioning of the vehicle 100, providing the vehicle 100's geographical location information. The vehicle speed sensor 12 is used to detect the vehicle speed of the vehicle 100. The inertial measurement unit 13 may include a combination of an accelerometer and a gyroscope, and is a device for measuring the angular rate and acceleration of the vehicle 100. For example, during the movement of the vehicle 100, the inertial measurement unit can measure the changes in the vehicle's position and angle based on the vehicle's inertial acceleration, such as measuring the vehicle's acceleration and angular rate.

[0097] The control system 20 may include a steering unit 21, a braking unit 22, etc. The steering unit 21 may represent a system for adjusting the direction of travel of the vehicle 100, and may include, but is not limited to, a steering wheel or other structural devices for adjusting or controlling the direction of travel of the vehicle 100. The braking unit 22 may represent a system for slowing down the vehicle 100, and may also be referred to as the vehicle 100 braking system. It may include, but is not limited to, a brake controller, a reducer, or other structural devices for slowing down the vehicle 100. In practical applications, the braking unit 22 may utilize friction to slow down the tires of the vehicle 100, thereby slowing down the vehicle 100's speed.

[0098] Peripheral device 30 may include several components, such as the communication system 31, touch screen 32, user interface 33, etc., as shown in the figure. The communication system 31 is used to enable network communication between vehicle 100 and other devices besides vehicle 100. In practical applications, the communication system 31 can employ wireless communication technology or wired communication technology to achieve network communication between vehicle 100 and other devices. This wired communication technology can refer to communication between vehicle 100 and other devices via network cable or fiber optic cable, etc. This wireless communication technology includes, but is not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technology, etc. The touchscreen 32 can be used to detect operation commands displayed on the touchscreen 32. For example, the user can perform touch operations on the content data displayed on the touchscreen 32 according to actual needs to achieve the corresponding function, such as playing music, video, or other multimedia files. User interface 33 may specifically be a touch panel for detecting operation commands on the touch panel. User interface 33 may also be a physical button or a mouse. User interface 33 may also be a display screen for outputting data and displaying images or data. Optionally, user interface 33 may also be at least one device belonging to the category of peripheral devices, such as a touch screen, microphone, and speaker.

[0099] Several functions of the vehicle 100 are controlled and implemented by the computer system 50. The computer system 50 may include multiple processors such as a general-purpose processor 51, a CDC 52, an MDC 53, a T-BOX 54, as well as a memory 55 (also referred to as a storage device) and a gateway 56.

[0100] In practical applications, the memory 55 can be located either inside or outside the computer system 50, such as as a cache in the vehicle 100; this application does not impose any limitations on this. The general-purpose processor 51 can be, for example, a graphics processing unit (GPU). The general-purpose processor 51, CDC 52, MDC 53, and T-BOX 54 can be used to run relevant programs or corresponding instructions stored in the memory 55 to implement the corresponding functions of the vehicle 100.

[0101] In some implementations, the processor 51 can acquire training sample data, which includes sample images, and the sample images include target obstacles; in the labeled feature space corresponding to the sample images, the region corresponding to the target obstacles is enlarged to obtain an adjusted labeled feature space; the sample images are input into the perception model to be trained to obtain the perception feature space of the sample images; based on the difference between the perception feature space and the adjusted labeled feature space, the model parameters of the perception model are adjusted.

[0102] In some implementations, the processor 51 can acquire at least one image data collected by the vehicle; the at least one image data includes at least one target obstacle; based on the at least one image data, a perception feature space is predicted using a perception model, the perception feature space including the perception result of the target obstacle, wherein the perception model is obtained according to the model training method provided in the embodiments of this application; based on the perception result of the target obstacle, the recognition result of the target obstacle is obtained. Thus, in a purely visual scenario, the error in the perception result of the vehicle 100 based on the perception model can be reduced, thereby minimizing the possibility of misidentification and missed identification of obstacles by the vehicle 100.

[0103] The memory 55 may include volatile memory, such as RAM; it may also include non-volatile memory, such as ROM, flash memory, HDD, or SSD; or it may include a combination of the above types of memory. The memory 55 can be used to store a set of program code or instructions corresponding to program code, so that the general-purpose processor 51 can call the program code or instructions stored in the memory 55 to implement the corresponding functions of the vehicle 100. This function includes, but is not limited to, […]. Figure 9The schematic diagram of the functional framework of vehicle 100 shown includes some or all of the functions. In this application, memory 55 can store a set of program code for controlling vehicle 100. General-purpose processor 51, CDC 52, MDC 53, and T-BOX 54 can call this program code to control vehicle 100 to perform vehicle network switching.

[0104] Optionally, in addition to storing program code or instructions, memory 55 may also store road maps, driving lines, or other structural devices used to adjust or control the direction of travel of vehicle 100.

[0105] It should be noted that the above Figure 9 This is merely a schematic diagram of one possible functional framework for vehicle 100. In practical applications, vehicle 100 may include more or fewer systems or components, and this application is not limiting. Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0106] It should be understood that Figure 9 The structure of the vehicle 100 shown is only an example. In other embodiments, the vehicle 100 may include more or fewer modules, which is not limited herein.

[0107] This application also provides a program product that, when executed on a device, enables the device to implement the methods provided in the foregoing embodiments. For example, it can implement... Figure 4 The model training method shown, or its implementation Figure 7 The aforementioned prediction method.

[0108] This application also provides a readable storage medium storing one or more programs. When these programs are executed by a device, they cause the device to implement the methods provided in the foregoing embodiments. For example, implementing... Figure 4 The model training method shown, or its implementation Figure 7 The aforementioned prediction method.

[0109] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0110] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0111] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0112] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon by one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0113] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0114] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0115] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0116] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A model training method applied to electronic devices, characterized in that, The method includes: Acquire training sample data, wherein the training sample data includes sample images, the sample images include target obstacles, and the sample images are images captured by the vehicle; In the labeled feature space corresponding to the sample image, the region corresponding to the target obstacle is expanded according to the distance between the vehicle and the target obstacle to obtain the adjusted labeled feature space; The sample image is input into the perceptual model to be trained to obtain the perceptual feature space of the sample image; Based on the difference between the perceived feature space and the adjusted labeled feature space, the model parameters of the perceived model are adjusted. Wherein, the labeled feature space corresponding to the sample image and the perceptual feature space are voxel spaces, the region corresponding to the target obstacle is the region occupied by the labeled voxel set of the target obstacle, and the labeled voxel set includes M labeled voxels, where M is a positive integer greater than or equal to 1; Expanding the area corresponding to the target obstacle includes: In the voxels of the labeled feature space, determine the extended voxel of each of the M labeled voxels; The unlabeled extended voxels in the extended voxels of the i-th labeled voxel are labeled to obtain the adjusted labeled feature space, where i is a positive integer taking values ​​from 1 to M. Wherein, the extended voxel of the i-th labeled voxel is: a voxel in the labeled feature space whose distance from the i-th labeled voxel is less than the i-th first distance and which is located on one side of the i-th labeled voxel along the i-th expansion direction; Furthermore, the position of the vehicle projected into the labeled feature space is the first position; the i-th expansion direction is: the direction from the first position to the position of the i-th labeled voxel, or the direction from the first position projected onto the horizontal plane to the position of the i-th labeled voxel projected onto the horizontal plane.

2. The model training method according to claim 1, characterized in that, The step of annotating the unannotated extended voxels in the extended voxels of the i-th labeled voxel includes: Change the attributes of the unlabeled extended voxels in the extended voxels of the i-th labeled voxel to the same attributes as the i-th labeled voxel, wherein the attributes include: voxel occupancy status and obstacle category occupying the voxel.

3. The model training method according to claim 1, characterized in that, The i-th first distance is: the distance between the first position and the position of the i-th labeled voxel, reduced according to a first coefficient; or; The i-th first distance is the distance between the position of the first location projected onto the horizontal plane and the position of the i-th labeled voxel projected onto the horizontal plane, after being reduced according to the second coefficient.

4. The model training method according to claim 1, characterized in that, The step of adjusting the model parameters of the perception model based on the difference between the perceived feature space and the adjusted labeled feature space includes: Calculate the deviation between the predicted occupancy probability of voxels in the perceptual feature space and the occupancy probability of voxels in the adjusted labeled feature space; Based on the deviation, the model parameters of the perception model are adjusted.

5. The model training method according to any one of claims 1 to 4, characterized in that, The perception model includes an occupation grid prediction model.

6. An electronic device, characterized in that, include: Memory, used to store instructions; At least one processor is configured to execute the instructions to cause the electronic device to implement the model training method of any one of claims 1-5.

7. A vehicle, characterized in that, The vehicle stores instructions that are executed on the vehicle to enable the vehicle to implement the model training method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the model training method according to any one of claims 1-5.

9. A computer program product, characterized in that, When the computer program product is run on the device, it causes the device to execute the model training method according to any one of claims 1-5.

Images