Image information enhancement method and apparatus, model training method and apparatus, and device and medium

Abstract

Provided in the embodiments of the present application is an image information enhancement method. The method comprises: acquiring image information sequences which are collected by a plurality of cameras, and point cloud information; selecting an alternative target sample from a target sample set; adding the alternative target sample to image information at the current moment, so as to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the alternative target sample; and adding the alternative target sample to image information at other moments, so as to obtain enhanced image information at the other moments, and adding the point cloud information corresponding to the alternative target sample, such that the positions of the alternative target sample in the image information at the other moments and the image information at the current moment meet timing training requirements. In this way, by means of performing extraction, screening and mapping processing on target samples, the diversity of the target samples in image information is greatly increased, the number of parameters of a model is prevented from being too large, and the number distribution of the target samples is more balanced, thereby improving the generalization performance and the detection precision of the model, and also improving the detection performance for a relatively small number of target sample categories.

Description

Image information enhancement method, model training method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on June 13, 2023, with application number 202310692621.X, and application name “Image Information Enhancement Method, Model Training Method, Device, Equipment and Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to an image information enhancement method, a target detection model training method based on image information enhancement, an image information enhancement device, a target detection model training device based on image information enhancement, an electronic device, and a non-volatile readable storage medium. Background Art

[0004] The rapid development of artificial intelligence (AI) has brought about a leap forward in autonomous driving technology. Autonomous driving systems consist of three components: perception, planning, and control. The perception component, acting as the "eyes" of autonomous vehicles, plays a crucial role. The performance of an autonomous driving system is often largely determined by the quality of its perception system. The autonomous driving perception system refers to the autonomous vehicle's ability to perceive its surroundings through various sensors, such as cameras and radar. It must not only accurately identify vehicles, pedestrians, obstacles, and traffic signs, but also accurately identify, locate, and predict their speed. Considering the cost of mass-produced vehicles, cameras have been widely used in autonomous vehicles due to their low price, mature technology, and the rich surface information their images can capture. Three-dimensional object detection technology based on camera image data has been a research focus.

[0005] The perception system reads the camera image information and outputs information such as the position, length, width, height, and speed of the detected target in three-dimensional space, so that the driving vehicle can make further decisions and plans. The current three-dimensional target detection technology based on image data first detects targets such as vehicles, pedestrians, and bicycles in the image through existing image target detection technology, and obtains information such as the position and size of the target in the image coordinate system. Then, through the relationship between the camera and the world coordinate system, the detected targets are mapped to the world coordinate system to obtain information such as the position, length, width, height, and speed of each target in three-dimensional space. This method benefits from the existing mature two-dimensional target detection system and can output relatively accurate image target detection results. However, this two-step detection framework cannot achieve end-to-end training and is difficult to integrate and debug with subsequent modules of the autonomous driving system. At the same time, the target result mapping process has weak adjustment capabilities, resulting in poor three-dimensional accuracy.

[0006] During the training process of deep neural networks, overfitting often occurs due to the large number of model parameters, which affects the model's generalization performance. Furthermore, the uneven distribution of target sample types in the dataset can affect the detector's detection performance for categories with fewer sample types.

[0007] Summary of the Invention

[0008] The technical problem to be solved by the embodiments of the present application is to provide a target detection model training method, device, electronic device and non-volatile readable storage medium, so as to solve the problem that overfitting often occurs due to the excessive number of model parameters, thereby affecting the generalization performance of the model. At the same time, due to the uneven distribution of the number of target type samples in the data set, the detection performance of the detector for fewer sample categories will be affected.

[0009] In order to solve the above problems, the present application provides an image information enhancement method, which includes:

[0010] Obtaining image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0011] Select alternative target samples from the target sample set;

[0012] According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment;

[0013] According to the coordinate system related information of the current moment and other moments, the alternative target samples are added to the image information of other moments to obtain enhanced image information of other moments, and the point cloud information corresponding to the alternative target samples is added to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information of other moments relative to the position in the image information of the current moment meets the requirements of temporal training.

[0014] Optionally, according to the image information at the current moment, the candidate target sample is added to the image information at the current moment to obtain the enhanced image information at the current moment, including:

[0015] According to the three-dimensional labels of the current moment image information and the alternative target samples in the image information sequence, as well as the coordinate system related information, the alternative target samples and the target samples in the current moment image information are added to the current moment image information in order of depth from far to near, so as to obtain enhanced current moment image information.

[0016] Optionally, based on the coordinate system related information of the current moment and other moments, the candidate target sample is added to the image information of other moments, and the enhanced image information of other moments is obtained, including:

[0017] Based on the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, the alternative target samples and the target samples in the image information of other moments are added to the image information of other moments in order of depth from far to near to obtain enhanced image information of other moments; wherein the three-dimensional labels include speed information.

[0018] Optionally, before adding the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment, the method further includes:

[0019] Determine the current moment image information to which the candidate target sample should be added and the camera that captured the current moment image information based on the current moment image information and the three-dimensional label of the candidate target sample, as well as information related to the coordinate system;

[0020] Determine, based on the coordinate system related information at the current moment and other moments and the three-dimensional labels of the candidate target samples, the image information at other moments to which the candidate target samples should be added, and the camera that captures the image information at other moments;

[0021] If the camera that collects image information at other moments is the same as the camera that collects image information at the current moment, the candidate target samples are retained.

[0022] This application also provides a method for training an object detection model based on image information enhancement, the method comprising:

[0023] Obtaining image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0024] Select alternative target samples from the target sample set;

[0025] According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment;

[0026] According to the coordinate system related information of the current moment and other moments, the candidate target sample is added to the image information of the other moments to obtain enhanced image information of the other moments, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the other moments, so that the position of the candidate target sample in the image information of the other moments relative to the position in the image information of the current moment meets the requirements of temporal training;

[0027] The target detection model is trained by enhancing the image information at the current moment and the image information at other moments, as well as the target sample labeling data and point cloud information; the target detection results output by the target detection model include the target's location information, length, width, height and speed.

[0028] Optionally, the target detection model includes an image feature extraction module, a feature map depth estimation module, an image feature mapping module, and a target detection module. The target detection model is trained by using enhanced image information at the current moment and enhanced image information at other moments, target sample labeling data, and point cloud information. The training includes:

[0029] Input the enhanced current moment image information and enhanced images at other moments into the image feature extraction module for feature extraction to obtain corresponding image features, and output the image features to the feature map depth estimation module;

[0030] The image features and the depth map output by the feature map depth estimation module are input into the image feature mapping module to obtain the corresponding bird's-eye view features, and the bird's-eye view features are output to the target detection module.

[0031] Optionally, the feature map depth estimation module includes a first layer neural network and a second layer neural network. Before inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module, the method further includes:

[0032] Output the image features to the first layer of the neural network to obtain a first depth map output by the first layer of the neural network;

[0033] The first depth map and the feature data output by the first layer of the neural network are cascaded in the feature channel and input into the second layer of the neural network to obtain the second depth map output by the second layer of the neural network as the depth map output by the feature map depth estimation module.

[0034] Optionally, the target detection model further includes a temporal information fusion module, and the target detection result includes the speed of the target; inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module includes:

[0035] For the image information at each moment, the image features, the depth map output by the feature map depth estimation module, and the coordinate system related information at different moments are input into the image feature mapping module to obtain the corresponding bird's-eye view features;

[0036] The bird's-eye view features corresponding to each moment are input into the temporal information fusion module for fusion to obtain fused bird's-eye view features, and the fused bird's-eye view features are output to the target detection module.

[0037] Optionally, the target detection model further includes a feature decoding module, and outputting the bird's-eye view features to the target detection module includes:

[0038] Input the bird's-eye view features into the feature decoding module for decoding to obtain decoded bird's-eye view features;

[0039] The decoded bird's-eye view features are output to the target detection module.

[0040] The present application also provides an image information enhancement device, comprising:

[0041] An information acquisition module is used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0042] A sample selection module is used to select candidate target samples from the target sample set;

[0043] A first information enhancement module is configured to add candidate target samples to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and to add point cloud information corresponding to the candidate target samples to the point cloud information corresponding to the current moment;

[0044] The second information enhancement module is used to add the alternative target samples to the image information at other moments according to the coordinate system related information of the current moment and other moments, so as to obtain enhanced image information at other moments, and add the point cloud information corresponding to the alternative target samples to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training.

[0045] Optionally, the first information enhancement module includes:

[0046] The first sample adding submodule is used to add the alternative target samples and the target samples in the current moment image information to the current moment image information in the image information sequence, in order of depth from far to near, based on the three-dimensional labels of the alternative target samples and the coordinate system related information, to obtain enhanced current moment image information.

[0047] Optionally, the second information enhancement module includes:

[0048] The second sample adding submodule is used to add the alternative target samples and the target samples in the image information of other moments in order of depth from far to near based on the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, so as to enhance the image information of other moments; wherein the three-dimensional labels include speed information.

[0049] Optionally, the device further comprises:

[0050] a first camera determination module configured to determine, based on the current moment image information and the three-dimensional label of the candidate target sample, as well as information related to the coordinate system, the image information at the current moment to which the candidate target sample should be added, and a camera that captured the image information at the current moment, before adding the candidate target sample to the image information at the current moment to obtain enhanced image information at the current moment and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment;

[0051] A second camera determination module is used to determine the image information at other moments to which the candidate target sample should be added, and the camera that captures the image information at other moments, based on the coordinate system related information at the current moment and other moments and the three-dimensional label of the candidate target sample;

[0052] The sample retention module is used to retain the candidate target sample if the camera that collects the image information at other moments and the camera that collects the image information at the current moment are the same camera.

[0053] This application also provides a target detection model training device based on image information enhancement, the device comprising:

[0054] An information acquisition module is used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0055] A sample selection module is used to select candidate target samples from the target sample set;

[0056] A first information enhancement module is configured to add candidate target samples to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and to add point cloud information corresponding to the candidate target samples to the point cloud information corresponding to the current moment;

[0057] The second information enhancement module is used to add the alternative target samples to the image information at other moments according to the coordinate system related information of the current moment and other moments, so as to obtain enhanced image information at other moments, and add the point cloud information corresponding to the alternative target samples to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training.

[0058] Optionally, the first information enhancement module includes:

[0059] The first sample adding submodule is used to add the alternative target samples and the target samples in the current moment image information to the current moment image information in the image information sequence, in order of depth from far to near, based on the three-dimensional labels of the alternative target samples and the coordinate system related information, to obtain enhanced current moment image information.

[0060] Optionally, the second information enhancement module includes:

[0061] The second sample adding submodule is used to add the alternative target samples and the target samples in the image information of other moments in order of depth from far to near based on the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, so as to enhance the image information of other moments; wherein the three-dimensional labels include speed information.

[0062] Optionally, the device further comprises:

[0063] a first camera determination module configured to determine, based on the current moment image information and the three-dimensional label of the candidate target sample, as well as information related to the coordinate system, the image information at the current moment to which the candidate target sample should be added, and a camera that captured the image information at the current moment, before adding the candidate target sample to the image information at the current moment to obtain enhanced image information at the current moment and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment;

[0064] A second camera determination module is used to determine the image information at other moments to which the candidate target sample should be added, and the camera that captures the image information at other moments, based on the coordinate system related information at the current moment and other moments and the three-dimensional label of the candidate target sample;

[0065] The sample retention module is used to retain the candidate target sample if the camera that collects the image information at other moments and the camera that collects the image information at the current moment are the same camera.

[0066] The embodiment of the present application further discloses an electronic device, characterized in that it includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0067] Memory for storing computer programs;

[0068] The processor is used to implement the above method steps when executing the program stored in the memory.

[0069] The embodiments of the present application also disclose a non-volatile readable storage medium. When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute one or more methods in the embodiments of the present application.

[0070] According to an embodiment of the present application, by acquiring a sequence of image information and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments, an alternative target sample is selected from a target sample set, and according to the image information at the current moment, the alternative target sample is added to the image information at the current moment to obtain enhanced image information at the current moment, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to the current moment, and according to the coordinate system related information of the current moment and other moments, the alternative target sample is added to the image information at other moments to obtain enhanced image information at other moments, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to other moments, so that the position of the alternative target sample in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training, so that through target sample extraction, screening and mapping processing, the diversity of target samples in the image information is greatly increased, and the model parameters are avoided from being too large. At the same time, the number of target samples is more evenly distributed, which can improve the generalization performance and detection accuracy of the model, and can also improve the detection performance of fewer target sample categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] FIG1 shows a flowchart of a method for enhancing image information according to an embodiment of the present application;

[0072] FIG2 shows a flow chart of the target detection model;

[0073] FIG3 shows a flowchart of a method for training an object detection model based on image information enhancement according to an embodiment of the present application;

[0074] FIG4 shows a schematic diagram of the process of the depth estimation cascade;

[0075] FIG5 shows a structural block diagram of an embodiment of an image information enhancement device provided by an embodiment of the present application;

[0076] FIG6 shows a structural block diagram of an embodiment of a method and apparatus for training an object detection model based on image information enhancement provided by another embodiment of the present application;

[0077] Fig. 7 shows a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0079] 1 , a flowchart of a method for enhancing image information according to an embodiment of the present application is shown, which may include the following steps:

[0080] Step 101: Acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at a current moment and image information at other moments.

[0081] In the embodiments of the present application, multiple cameras capture images separately to obtain image information. For example, multiple cameras on a vehicle capture image information. Point cloud information refers to a collection of information about points in a certain coordinate system. Points contain a wealth of information, including three-dimensional coordinates X, Y, and Z, color, classification value, intensity value, time, and so on. For example, point cloud information can be obtained by lidar, which is not limited in the embodiments of the present application.

[0082] In the embodiment of the present application, the image information sequence includes the current moment image information and the image information of other moments. There can be multiple current moment image information. There can be multiple other moment image information. For example, as shown in FIG2, the current moment image information I t , image information at other times I t-1 .

[0083] Step 102: Select candidate target samples from the target sample set.

[0084] In the embodiment of the present application, the target can be a vehicle, person, animal, railing, telephone pole, etc. in the image, or any other applicable target, and the embodiment of the present application does not limit this. The target used to train the model is recorded as a target sample. The target sample labeling data includes the depth of the target, and the depth of the target refers to the distance between the target and the target. The target sample labeling data can also include the classification, length, width, height, speed, three-dimensional position, etc. of the target, or any other applicable data, and the embodiment of the present application does not limit this.

[0085] In an embodiment of the present application, target sample labeled data can be used as the true value of the target detection result during training, and point cloud information can be used to determine the true value of the depth map during training.

[0086] In the embodiment of the present application, the target sample set is a collection of targets and related data. The target sample set can be generated by combining the point cloud information and image information of the target in advance.

[0087] For example, the training data set includes but is not limited to image information at different times, point cloud information, coordinate system related information, and target 3D labels. From the entire training data set, according to the 3D label G of each target sample s, s Extract the corresponding point cloud information P s With image block I s Then, all the extracted target sample data (3D labels, point cloud information, image blocks) and the corresponding coordinate system related information are organized into a database, namely the target sample set.

[0088] The 3D label refers to the 3D information label of the target, including but not limited to the target's 3D position, length, width, height, and speed. Coordinate system related information includes but is not limited to the camera parameters, radar (point cloud information) parameters, and vehicle pose corresponding to the target sample at the time.

[0089] In an embodiment of the present application, during the model training process, candidate target samples are randomly selected from the target sample set, and sample screening and mapping are performed based on their three-dimensional spatial coordinate information to enhance the image data.

[0090] For example, during the training process, candidate target samples are selected from the target sample set to obtain the candidate target sample set. Specifically, the candidate target sample set can be obtained by random sampling or any other applicable method, which is not limited in the embodiments of the present application.

[0091] Step 103 , according to the current moment image information, the candidate target samples are added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target samples is added to the point cloud information corresponding to the current moment.

[0092] In an embodiment of the present application, in order to solve the problem of excessive model parameters and uneven distribution of the number of target samples in the data set, and to improve the generalization performance and detection accuracy of the model, in addition to using conventional image data enhancement methods such as scaling, cropping, rotation, and flipping, the present application proposes an image data enhancement technology based on the target sample set.

[0093] In the embodiments of the present application, a three-dimensional sample data enhancement algorithm is developed. In the data preprocessing stage, sample extraction, screening and mapping are performed, which greatly increases the diversity of targets and thus improves the target detection accuracy.

[0094] In the embodiment of the present application, based on the current moment image information, the candidate target samples are added to the current moment image information, thereby increasing the number of target samples in the current moment image information. The resulting new current moment image information is recorded as the enhanced current moment image information. Any suitable addition method can be used, and the embodiment of the present application does not limit this.

[0095] In an embodiment of the present application, since alternative target samples are added to the image information at the current moment, point cloud information corresponding to the alternative target samples should also be added to the point cloud information corresponding to the current moment, so that the point cloud information can correspond to the enhanced image information at the current moment.

[0096] In an optional embodiment of the present application, according to the image information at the current moment, the alternative target samples are added to the image information at the current moment to obtain a specific implementation method of enhancing the image information at the current moment, which may include: according to the three-dimensional labels of the image information at the current moment and the alternative target samples in the image information sequence, as well as the coordinate system related information, the alternative target samples and the target samples in the image information at the current moment are added to the image information at the current moment in order of depth from far to near to obtain enhanced image information at the current moment.

[0097] Coordinate system related information refers to the information that describes the mutual conversion relationship between the vehicle's posture, vehicle coordinate system, camera coordinate system, radar (i.e., point cloud information) coordinate system, etc.

[0098] The original target samples in the current image also need to be re-added to the current image. The candidate target samples and the target samples in the current image are added to the current image in order of depth from far to near, so that the near target occludes the far target.

[0099] For example, first, based on the camera intrinsic parameters at the current moment, the external parameter relationship between the camera and the radar, and the radar truth frame of the alternative target sample, the camera index, image truth frame, and three-dimensional truth frame of the alternative target sample pasted to the image information at the current moment are obtained.

[0100] Among them, the radar truth frame is the position data of the alternative target sample in the radar (point cloud information) coordinate system determined based on the three-dimensional label of the alternative target sample and the positional relationship between the radar and the vehicle. The camera index refers to the current moment image information of a camera determined by the alternative target sample in the current moment image information of multiple cameras based on the three-dimensional label of the alternative target sample, so that the position of the alternative target sample relative to the radar coordinate system is different. The camera index is used to represent the camera. The image truth frame is the category, length, width, height, position and other data of the alternative target sample in the current moment image information. The three-dimensional truth frame is the three-dimensional label data of the alternative target sample in the current moment image information.

[0101] Then, the image truth frame, three-dimensional truth frame, image block and point cloud information corresponding to all target samples in the current image information are obtained.

[0102] Next, the candidate target samples and all target samples in the current image are mapped in descending order, based on the depth values ​​corresponding to the center point of the 3D ground truth box under each camera. This adds the candidate target samples and the target samples in the current image to the current image. The corresponding point cloud information is then pasted together. Ultimately, the enhanced current image and its corresponding point cloud are obtained.

[0103] Step 104, based on the coordinate system related information of the current moment and other moments, the alternative target sample is added to the image information of other moments to obtain enhanced image information of other moments, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to other moments, so that the position of the alternative target sample in the image information of other moments relative to the position in the image information of the current moment meets the requirements of temporal training.

[0104] In an embodiment of the present application, the coordinate system related information at different times includes information on the mutual conversion relationship between the vehicle's posture, vehicle coordinate system, camera coordinate system, and radar (i.e., point cloud information) coordinate system at different times.

[0105] In the embodiment of the present application, if alternative target samples are added to the image information at the current moment, then alternative target samples should also be added to the image information at other moments, so that the image information at different moments has coherence. To this end, it is necessary to add alternative target samples to the image information at other moments based on the coordinate system related information at the current moment and the coordinate system related information at other moments, and the new image information at other moments obtained is recorded as enhanced image information at other moments, so that the position of the alternative target samples in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training, that is, the position change of the alternative target samples in the image information at the current moment and the image information at other moments should be logically reasonable. For example, the position change of the alternative target samples in the image information at the current moment I t Position A, relative to the image information I at other times t-1 The distance between position A and position B can remain unchanged, or the distance can be increased by m, or the distance can be reduced by n, and m and n should be within a reasonable range.

[0106] In an embodiment of the present application, there may be multiple specific implementation methods for adding candidate target samples to image information at other moments based on coordinate system related information at the current moment and other moments, and the embodiment of the present application does not limit this.

[0107] In an embodiment of the present application, since alternative target samples are added to the image information at other moments, point cloud information corresponding to the alternative target samples should also be added to the point cloud information corresponding to other moments, so that the point cloud information can correspond to the enhanced image information at other moments.

[0108] In an optional embodiment of the present application, according to the coordinate system related information of the current moment and other moments, the alternative target samples are added to the image information of other moments to obtain enhanced image information of other moments. A specific implementation method may include: according to the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, the alternative target samples and the target samples in the image information of other moments are added to the image information of other moments in order of depth from far to near to obtain enhanced image information of other moments; wherein the three-dimensional labels include speed information.

[0109] The three-dimensional label of the alternative target sample includes speed information. According to the speed information, the distance between the position of the alternative target sample in the image information at the current moment and the position in the image information at other moments should be consistent with the speed information. That is to say, after the position of the alternative target sample in the image information at the current moment is determined, the position of the alternative target sample in the image information at other moments is determined according to the speed information and the time difference between different moments.

[0110] For example, first, based on the coordinate system information at the current moment and the other moments, and the 3D labels of the candidate target samples, the camera index, image ground truth box, and 3D ground truth box of the candidate target samples pasted into the image information at the other moments are obtained. Then, the image ground truth box, 3D ground truth box, image block, and point cloud information corresponding to all target samples in the image information at the other moments are obtained. Finally, all target samples in the candidate target samples and the image information at the other moments are mapped in order from far to near according to the depth values ​​corresponding to the center points of the 3D ground truth boxes under each camera. This allows the candidate target samples and the target samples in the other moment image information to be added to the image information at the other moments, ensuring that the positions of the candidate target samples in the image information at the other moments relative to the positions in the image information at the current moment meet the requirements of temporal training. The corresponding point cloud information is then pasted. Ultimately, the enhanced image information at the current moment and the corresponding point cloud information are obtained.

[0111] Among them, the requirement of temporal training is to ensure the consistency and accuracy of image information at different times. For example, the position of the target sample in the image information at different times should be consistent with the speed of the target sample.

[0112] In an optional embodiment of the present application, before adding the alternative target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to the current moment, it may also include: determining the image information at the current moment to which the alternative target sample should be added, and the camera that collects the image information at the current moment according to the image information at the current moment and the three-dimensional label of the alternative target sample, as well as coordinate system related information; determining the image information at other moments to which the alternative target sample should be added, and the camera that collects image information at other moments according to coordinate system related information at the current moment and other moments, and the three-dimensional label of the alternative target sample; if the camera that collects image information at other moments and the camera that collects image information at the current moment are the same camera, retaining the alternative target sample.

[0113] In order to ensure the consistency and accuracy of image information at different times, it is necessary to screen the candidate target samples. According to the RT (rotation and translation) relationship between the radar coordinate system at the current time and other times, and the speed of the candidate target sample j, the target sample j at other times is obtained. adj The corresponding three-dimensional true value (3D label data in radar coordinate system). Project it to the corresponding camera. If it is consistent with the camera index corresponding to the candidate target sample j at the current moment, the sample is retained.

[0114] According to an embodiment of the present application, by acquiring a sequence of image information and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments, an alternative target sample is selected from a target sample set, and according to the image information at the current moment, the alternative target sample is added to the image information at the current moment to obtain enhanced image information at the current moment, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to the current moment, and according to the coordinate system related information of the current moment and other moments, the alternative target sample is added to the image information at other moments to obtain enhanced image information at other moments, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to other moments, so that the position of the alternative target sample in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training, so that through target sample extraction, screening and mapping processing, the diversity of target samples in the image information is greatly increased, and the model parameters are avoided from being too large. At the same time, the number of target samples is more evenly distributed, which can improve the generalization performance and detection accuracy of the model, and can also improve the detection performance of fewer target sample categories.

[0115] 3 , a flowchart of a method for training an object detection model based on image information enhancement according to an embodiment of the present application is shown. The method may include the following steps:

[0116] Step 201 : Acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at a current moment and image information at other moments.

[0117] In the embodiment of the present application, the specific implementation method of this step can be found in the description of the aforementioned embodiment and will not be repeated here.

[0118] Step 202: Select candidate target samples from the target sample set.

[0119] In the embodiment of the present application, the specific implementation method of this step can be found in the description of the aforementioned embodiment and will not be repeated here.

[0120] Step 203 : According to the image information at the current moment, the candidate target samples are added to the image information at the current moment to obtain enhanced image information at the current moment, and the point cloud information corresponding to the candidate target samples is added to the point cloud information corresponding to the current moment.

[0121] In the embodiment of the present application, the specific implementation method of this step can be found in the description of the aforementioned embodiment and will not be repeated here.

[0122] In step 204, based on the coordinate system related information of the current moment and other moments, the alternative target samples are added to the image information of other moments to obtain enhanced image information of other moments, and the point cloud information corresponding to the alternative target samples is added to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information of other moments relative to the position in the image information of the current moment meets the requirements of temporal training.

[0123] In the embodiment of the present application, the specific implementation method of this step can be found in the description of the aforementioned embodiment and will not be repeated here.

[0124] In step 205, the target detection model is trained using the enhanced image information at the current moment and the enhanced image information at other moments, the target sample label data, and the point cloud information; wherein the target detection results output by the target detection model include the target's location information, length, width, height, and speed.

[0125] In the embodiments of the present application, the object detection model can detect the location information, length, width, height, classification, speed, etc. of the object in the image, or any other applicable detection, which is not limited by the embodiments of the present application. The location information can be a three-dimensional position, that is, in addition to the position on the image plane, it also includes the depth of the image.

[0126] In an embodiment of the present application, in order to enable the target detection model to detect targets in image information or target images at different times and output target detection results, it is necessary to use target sample labeling data labeled for image information to train the target detection model.

[0127] For example, the final loss function used during training is L = λ depth L depth +λ cls L cls +λ reg L reg . Where L depth ,L cls ,L reg They are respectively the depth estimation loss of image features, target classification loss, target position, length, width, height, and speed estimation loss. depth ,λ cls ,λ reg is the weight corresponding to each loss.

[0128] In the embodiment of the present application, the target detection model is trained using time series data. Specifically, the target detection model is trained using enhanced image information at the current moment and enhanced image information at other moments, target sample label data, and point cloud information.

[0129] In an optional embodiment of the present application, the target detection model includes an image feature extraction module, a feature map depth estimation module, an image feature mapping module and a target detection module.

[0130] A specific implementation method of training a target detection model by enhancing the image information at the current moment and the image information at other moments, and target sample labeling data and point cloud information can include: inputting the enhanced image information at the current moment and the enhanced images at other moments into an image feature extraction module for feature extraction to obtain corresponding image features, and outputting the image features to a feature map depth estimation module; inputting the image features and the depth map output by the feature map depth estimation module into an image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to a target detection module.

[0131] In the embodiment of the present application, the image feature extraction module is a module for feature extraction, and specifically can perform image feature extraction through a deep neural network. The specific implementation method of image feature extraction can adopt any applicable implementation method, and the embodiment of the present application does not limit this.

[0132] In the embodiment of the present application, the enhanced image information at the current moment is input into the image feature extraction module for feature extraction to obtain image features corresponding to the enhanced image information at the current moment. The enhanced images at other moments are input into the image feature extraction module for feature extraction to obtain image features corresponding to the enhanced image information at other moments.

[0133] For example, the process diagram of the target detection model is shown in Figure 2. The image feature extraction module obtains the image sequence I of multiple cameras. t ,I t-1 , I t is the image information at time t (6), I t-1 The image information at time t-1 (6 pieces) is extracted through the image feature extraction module to obtain the corresponding image features Where n represents the number of cameras, H F ,W F represents the scale of the image feature, and C represents the number of channels of the image feature.

[0134] In the embodiments of the present application, the feature map depth estimation module is a module for obtaining a depth map corresponding to an image feature map. Specifically, the depth map can be extracted using a multi-layer neural network. The true value of the depth map output by the feature map depth estimation module is determined by point cloud information. Any applicable implementation method can be used, and the embodiments of the present application are not limited thereto.

[0135] In the embodiment of the present application, the image features corresponding to the enhanced image information at the current moment are output to the feature map depth estimation module. In addition, the image features corresponding to the enhanced image information at other moments are output to the feature map depth estimation module.

[0136] For example, as shown in Figure 2, based on the point cloud information, a true depth map in the image space is generated, and then the generated true depth map is used to train the feature map depth estimation module. Get the depth map corresponding to the image feature map Among them, D is the number of depth quantization (that is, the number of depth value ranges in the depth map), that is, the specified depth [depth min ,depth max ] is divided into D units, and the value on the i-th unit represents the probability that the depth of the current feature point is within the depth range of the i-th unit.

[0137] In an optional embodiment of the present application, the object detection model further includes an image feature enhancement module. A specific implementation of outputting image features to the feature map depth estimation module may include: inputting the image features into the image feature enhancement module for feature enhancement to obtain enhanced image features; and outputting the enhanced image features to the feature map depth estimation module.

[0138] The image feature enhancement module is a module used to enhance image features. For example, by upsampling, the features of a large image can be effectively learned to obtain enhanced image features. Any applicable implementation method can be adopted, and the embodiments of the present application do not limit this.

[0139] For example, the image feature enhancement module is a multi-layer neural network that further improves the image feature expression capability and obtains enhanced image features.

[0140] In an embodiment of the present application, the bird's eye view (BEV) feature is feature data under the BEV view. The BEV view refers to the projection of the point cloud on a plane perpendicular to the height direction. Usually, before obtaining the BEV view, the space is divided into voxels, the point cloud is downsampled using voxels, and then each voxel is projected as a point. A voxel refers to a cuboid when the three-dimensional space is divided into cuboids of fixed size. The pixel coordinates of the BEV view can be obtained when the voxels are projected. The feature value of each pixel can be obtained in a variety of ways.

[0141] In an embodiment of the present application, the image feature mapping module is used to generate a model of bird's-eye view features based on image features and depth maps through coordinate system mapping. Any applicable implementation method can be adopted, and the embodiment of the present application does not limit this.

[0142] In an embodiment of the present application, the target detection module is a model for detecting targets in image information or target images at different times based on bird's-eye view features, and outputting target detection results. Any applicable implementation method can be adopted, and the embodiment of the present application does not limit this.

[0143] In an embodiment of the present application, the image features corresponding to the enhanced image information at the current moment, the image features corresponding to the enhanced image information at other moments, and the depth map output by the feature map depth estimation module are input into the image feature mapping module to obtain the corresponding bird's-eye view features, and the bird's-eye view features are output to the target detection module.

[0144] For example, as shown in Figure 2, according to the enhanced image feature F t ,F t-1 and the estimated depth map D t ,D t-1 , generate three-dimensional features in the image coordinate system Specifically, Each point feature in That is, the outer product of the two vectors. According to the camera intrinsic parameters, the rotation and translation relationship between the camera and the autonomous driving vehicle, the three-dimensional feature F in the image coordinate system is converted t ,F t-1 The features of each point in the image are converted to the autonomous vehicle coordinate system. The feature points converted to the autonomous vehicle coordinate system are voxelized to form a three-dimensional feature representation in the autonomous vehicle coordinate system. The feature points that fall into the same voxel grid are accumulated, and the features of the voxel grid where no feature points fall are set to all 0. Finally, along the height dimension, the features corresponding to the voxels at all heights are accumulated to obtain the final bird's-eye view features. Among them, X and Y represent the front and back and left and right dimensions of the bird's-eye view features.

[0145] In an optional embodiment of the present application, the feature map depth estimation module includes a first-layer neural network and a second-layer neural network. Before inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain the corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module, it can also include: outputting the image features to the first-layer neural network to obtain a first depth map output by the first-layer neural network; cascading the first depth map and the feature data output by the first-layer neural network in the feature channel, and inputting them into the second-layer neural network to obtain a second depth map output by the second-layer neural network as the depth map output by the feature map depth estimation module.

[0146] The first neural network layer takes image features as input and outputs a first depth map. The true value of the first depth map output by the first neural network layer is determined by the point cloud information. Any suitable implementation method may be used, and this embodiment of the application is not limited thereto.

[0147] The second neural network layer takes the first depth map and the feature data obtained by concatenating the feature data in the first neural network layer as input and outputs a second depth map. The true value of the second depth map output by the second neural network layer is determined by the point cloud information. Any suitable implementation method can be used, and this embodiment of the application is not limited thereto.

[0148] During the cascading process, this application uses point cloud information to supervise the training of the feature map depth estimation module to improve the accuracy of mapping image features into three-dimensional space.

[0149] For example, Figure 4 shows a flow chart of depth map estimation. To further improve the accuracy of depth estimation, a two-layer cascaded depth estimation network is used. The depth prediction image 1 estimated by the first layer of the neural network (denoted as the first depth map) is concatenated with the features output by the first layer of the neural network in the feature channel and input into the second layer of the neural network. The output depth prediction image 2 (denoted as the second depth map) serves as the depth map used by subsequent modules. In this way, the accuracy of depth estimation is improved.

[0150] In an optional embodiment of the present application, the target detection model further includes a feature decoding module. A specific implementation of outputting the bird's-eye view features to the target detection module may include: inputting the bird's-eye view features into the feature decoding module for decoding to obtain decoded bird's-eye view features; and outputting the decoded bird's-eye view features to the target detection module.

[0151] The feature decoding module is used to decode the bird's-eye view features and obtain the decoded bird's-eye view features. For example, as shown in Figure 2, the fused BEV features are decoded by a multi-layer neural convolutional network to obtain the final bird's-eye view feature output. Specifically, any applicable implementation method can be adopted, and the embodiments of the present application do not limit this.

[0152] In an optional embodiment of the present application, the target detection model also includes a temporal information fusion module, and the target detection result includes the speed of the target. The image features and the depth map output by the feature map depth estimation module are input into the image feature mapping module to obtain the corresponding bird's-eye view features, and the bird's-eye view features are output to the target detection module, including: for the image information at each moment, the image features, the depth map output by the feature map depth estimation module and the coordinate system related information at different moments are input into the image feature mapping module to obtain the corresponding bird's-eye view features; the bird's-eye view features corresponding to each moment are input into the temporal information fusion module for fusion to obtain the fused bird's-eye view features, and the fused bird's-eye view features are output to the target detection module.

[0153] Single-frame images have limited information, which affects the detection performance of three-dimensional objects. This application introduces a four-dimensional space formed by temporal fusion of time information, which not only greatly improves the prediction accuracy of three-dimensional object speed, but also achieves better detection of temporarily occluded objects.

[0154] The temporal information fusion module is used to fuse bird's-eye view features from different moments. It generates corresponding bird's-eye view features for each moment's image information. These features are then fused together.

[0155] The image features, the depth map output by the feature map depth estimation module, and the coordinate system related information at different moments are respectively input into the image feature mapping module to obtain the corresponding bird's-eye view features. For example, to enhance the image information at the current moment, the corresponding image features, the depth map output by the feature map depth estimation module, and the coordinate system related information are input into the image feature mapping module to obtain the bird's-eye view features corresponding to the enhanced image information at the current moment. To enhance image information at other moments, the corresponding image features, the depth map output by the feature map depth estimation module, and the coordinate system related information are input into the image feature mapping module to obtain the bird's-eye view features corresponding to the enhanced image information at other moments.

[0156] Then, the bird's-eye view features corresponding to each moment are input into the temporal information fusion module for fusion to obtain the fused bird's-eye view features.

[0157] For example, as shown in Figure 2, the BEV characteristics at time t and time t-1 are The fused bird's-eye view features are input into the time series information fusion module for fusion. The fused bird's-eye view features are recorded as fused bird's-eye view features. The fused bird's-eye view features are output to the object detection module.

[0158] For example, as shown in Figure 2 and The bird's-eye view features corresponding to each moment are input into the temporal information fusion module for fusion to obtain fused bird's-eye view features, thereby further reducing the impact of the collection vehicle's own movement on the alignment of bird's-eye view features at different moments.

[0159] In addition, during the training process, random frames are extracted to increase the diversity of temporal information. Image information from multiple adjacent moments is randomly selected as the image information of the previous moment and matched with the image information of the current moment, which greatly enhances the diversity of temporal information.

[0160] In the present embodiment, the object detection model trained as described above is used as input to a target image sequence captured by multiple cameras. The target detection model performs target detection, thereby obtaining target detection results. The target detection model can detect the depth of the target, as well as the target's classification, length, width, height, speed, three-dimensional position, and other information.

[0161] According to an embodiment of the present application, by acquiring an image information sequence and corresponding point cloud information acquired by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments, selecting an alternative target sample from a target sample set, adding the alternative target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to the current moment, adding the alternative target sample to the image information at other moments according to coordinate system related information at the current moment and other moments to obtain enhanced image information at other moments, and adding the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to other moments, In order to make the position of the alternative target sample in the image information at other times relative to the position in the image information at the current time meet the requirements of temporal training, the target detection model is trained by enhancing the image information at the current time and enhancing the image information at other times, and the target sample labeling data and point cloud information; wherein, the target detection result output by the target detection model includes the position information, length, width, height and speed of the target, so that the diversity of target samples in the image information is greatly increased through target sample extraction, screening and mapping processing, avoiding the target detection model from having too many parameters. At the same time, the number of target samples is more evenly distributed, which can improve the generalization performance and detection accuracy of the target detection model, and can also improve the detection performance of fewer target sample categories.

[0162] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0163] 5 , a structural block diagram of an image information enhancement device according to another embodiment of the present application is shown, which may include the following modules:

[0164] The information acquisition module 301 is used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequences include image information at the current moment and image information at other moments;

[0165] The sample selection module 302 is used to select candidate target samples from the target sample set;

[0166] The first information enhancement module 303 is configured to add candidate target samples to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and to add point cloud information corresponding to the candidate target samples to the point cloud information corresponding to the current moment;

[0167] The second information enhancement module 304 is used to add the alternative target samples to the image information at other moments according to the coordinate system related information of the current moment and other moments, so as to obtain enhanced image information at other moments, and add the point cloud information corresponding to the alternative target samples to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information at other moments relative to the position in the image information at the current moment meets the requirements of time series training.

[0168] Optionally, the first information enhancement module includes:

[0169] The first sample adding submodule is used to add the alternative target samples and the target samples in the current moment image information to the current moment image information in the image information sequence, in order of depth from far to near, based on the three-dimensional labels of the alternative target samples and the coordinate system related information, to obtain enhanced current moment image information.

[0170] Optionally, the second information enhancement module includes:

[0171] The second sample adding submodule is used to add the alternative target samples and the target samples in the image information of other moments in order of depth from far to near based on the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, so as to enhance the image information of other moments; wherein the three-dimensional labels include speed information.

[0172] Optionally, the device further comprises:

[0173] a first camera determination module configured to determine, based on the current moment image information and the three-dimensional label of the candidate target sample, as well as information related to the coordinate system, the image information at the current moment to which the candidate target sample should be added, and a camera that captured the image information at the current moment, before adding the candidate target sample to the image information at the current moment to obtain enhanced image information at the current moment and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment;

[0174] A second camera determination module is used to determine the image information at other moments to which the candidate target sample should be added, and the camera that captures the image information at other moments, based on the coordinate system related information at the current moment and other moments and the three-dimensional label of the candidate target sample;

[0175] The sample retention module is used to retain the candidate target sample if the camera that collects the image information at other moments and the camera that collects the image information at the current moment are the same camera.

[0176] According to an embodiment of the present application, by acquiring a sequence of image information and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments, an alternative target sample is selected from a target sample set, and according to the image information at the current moment, the alternative target sample is added to the image information at the current moment to obtain enhanced image information at the current moment, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to the current moment, and according to the coordinate system related information of the current moment and other moments, the alternative target sample is added to the image information at other moments to obtain enhanced image information at other moments, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to other moments, so that the position of the alternative target sample in the image information at other moments relative to the position in the image information at the current moment meets the requirements of temporal training, so that through target sample extraction, screening and mapping processing, the diversity of target samples in the image information is greatly increased, and the model parameters are avoided from being too large. At the same time, the number of target samples is more evenly distributed, which can improve the generalization performance and detection accuracy of the model, and can also improve the detection performance of fewer target sample categories.

[0177] 6 , a block diagram of an embodiment of an object detection device based on an object detection model according to another embodiment of the present application is shown, which may include the following modules:

[0178] The information acquisition module 401 is used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequences include image information at the current moment and image information at other moments;

[0179] The sample selection module 402 is used to select candidate target samples from the target sample set;

[0180] The first sample adding module 403 is used to add the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and add the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment;

[0181] The second sample adding module 404 is configured to add candidate target samples to the image information at other moments based on the coordinate system related information of the current moment and other moments to obtain enhanced image information at other moments, and to add the point cloud information corresponding to the candidate target samples to the point cloud information corresponding to the other moments so that the position of the candidate target samples in the image information at other moments relative to the position in the image information at the current moment meets the requirements of time series training;

[0182] The model training module 405 is used to train the target detection model by using enhanced image information at the current moment and enhanced image information at other moments, target sample labeling data and point cloud information; wherein the target detection results output by the target detection module include the target's location information, length, width, height and speed.

[0183] Optionally, the target detection model includes an image feature extraction module, a feature map depth estimation module, an image feature mapping module, and a target detection module, and the model training module includes:

[0184] The feature extraction submodule is used to input the enhanced image information at the current moment and the enhanced images at other moments into the image feature extraction module for feature extraction, obtain the corresponding image features, and output the image features to the feature map depth estimation module;

[0185] The feature output submodule is used to input the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain the corresponding bird's-eye view features, and output the bird's-eye view features to the target detection module.

[0186] Optionally, the feature map depth estimation module includes a first layer neural network and a second layer neural network, and the apparatus further includes:

[0187] A first depth map generation submodule is configured to input the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and output the image features to the first layer of the neural network before outputting the bird's-eye view features to the target detection module to obtain a first depth map output by the first layer of the neural network;

[0188] The second depth map generation submodule is used to input the first depth map and the feature data output by the first layer of the neural network into the second layer of the neural network after cascading the feature channel, so as to obtain the second depth map output by the second layer of the neural network as the depth map output by the feature map depth estimation module.

[0189] Optionally, the target detection model further includes a temporal information fusion module, and the target detection result includes the target speed; the feature output submodule includes:

[0190] An information input unit is used to input the image features, the depth map output by the feature map depth estimation module, and the coordinate system related information at different times into the image feature mapping module for the image information at each moment, so as to obtain the corresponding bird's-eye view features;

[0191] The feature output unit is used to input the bird's-eye view features corresponding to each moment into the time series information fusion module for fusion, obtain the fused bird's-eye view features, and output the fused bird's-eye view features to the target detection module.

[0192] Optionally, the target detection model further includes a feature decoding module, and the feature output submodule includes:

[0193] A decoding unit, configured to input the bird's-eye view features into a feature decoding module for decoding to obtain decoded bird's-eye view features;

[0194] The output unit is used to output the decoded bird's-eye view features to the target detection module.

[0195] According to an embodiment of the present application, by acquiring an image information sequence and corresponding point cloud information acquired by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments, selecting an alternative target sample from a target sample set, adding the alternative target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to the current moment, adding the alternative target sample to the image information at other moments according to coordinate system related information at the current moment and other moments to obtain enhanced image information at other moments, and adding the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to other moments, In order to make the position of the alternative target sample in the image information at other times relative to the position in the image information at the current time meet the requirements of temporal training, the target detection model is trained by enhancing the image information at the current time and enhancing the image information at other times, and the target sample labeling data and point cloud information; wherein, the target detection result output by the target detection model includes the position information, length, width, height and speed of the target, so that the diversity of target samples in the image information is greatly increased through target sample extraction, screening and mapping processing, avoiding the target detection model from having too many parameters. At the same time, the number of target samples is more evenly distributed, which can improve the generalization performance and detection accuracy of the target detection model, and can also improve the detection performance of fewer target sample categories.

[0196] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0198] 7 , electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0199] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 702 may include one or more modules to facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.

[0200] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0201] The power component 704 provides power to the various components of the electronic device 700. The power component 704 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0202] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0203] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0204] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0205] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect changes in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and temperature changes of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0206] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 714 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 714 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0207] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0208] In an exemplary embodiment, a non-transitory computer-readable non-volatile storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the electronic device 700 to perform the above method. For example, the non-transitory computer-readable non-volatile storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] A non-transitory computer-readable non-volatile storage medium, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform an image information enhancement method, the method comprising:

[0210] Obtaining image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0211] Select alternative target samples from the target sample set;

[0212] According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment;

[0213] According to the coordinate system related information of the current moment and other moments, the alternative target samples are added to the image information of other moments to obtain enhanced image information of other moments, and the point cloud information corresponding to the alternative target samples is added to the point cloud information corresponding to other moments, so that the position of the alternative target samples in the image information of other moments relative to the position in the image information of the current moment meets the requirements of temporal training.

[0214] Optionally, according to the image information at the current moment, the candidate target sample is added to the image information at the current moment to obtain the enhanced image information at the current moment, including:

[0215] According to the three-dimensional labels of the current moment image information and the alternative target samples in the image information sequence, as well as the coordinate system related information, the alternative target samples and the target samples in the current moment image information are added to the current moment image information in order of depth from far to near, so as to obtain enhanced current moment image information.

[0216] Optionally, based on the coordinate system related information of the current moment and other moments, the candidate target sample is added to the image information of other moments, and the enhanced image information of other moments is obtained, including:

[0217] Based on the coordinate system related information of the current moment and other moments and the three-dimensional labels of the alternative target samples, the alternative target samples and the target samples in the image information of other moments are added to the image information of other moments in order of depth from far to near to obtain enhanced image information of other moments; wherein the three-dimensional labels include speed information.

[0218] Optionally, before adding the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment, the method further includes:

[0219] Determine the current moment image information to which the candidate target sample should be added and the camera that captured the current moment image information based on the current moment image information and the three-dimensional label of the candidate target sample, as well as information related to the coordinate system;

[0220] Determine, based on the coordinate system related information at the current moment and other moments and the three-dimensional labels of the candidate target samples, the image information at other moments to which the candidate target samples should be added, and the camera that captures the image information at other moments;

[0221] If the camera that collects image information at other moments is the same as the camera that collects image information at the current moment, the candidate target samples are retained.

[0222] A non-transitory computer-readable non-volatile storage medium, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform a method for training an object detection model based on image information enhancement, the method comprising:

[0223] Obtaining image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments;

[0224] Select alternative target samples from the target sample set;

[0225] According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment;

[0226] According to the coordinate system related information of the current moment and other moments, the candidate target sample is added to the image information of the other moments to obtain enhanced image information of the other moments, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the other moments, so that the position of the candidate target sample in the image information of the other moments relative to the position in the image information of the current moment meets the requirements of temporal training;

[0227] The target detection model is trained by enhancing the image information at the current moment and the image information at other moments, as well as the target sample labeling data and point cloud information; the target detection results output by the target detection model include the target's location information, length, width, height and speed.

[0228] Optionally, the target detection model includes an image feature extraction module, a feature map depth estimation module, an image feature mapping module, and a target detection module. The target detection model is trained by using enhanced image information at the current moment and enhanced image information at other moments, target sample labeling data, and point cloud information. The training includes:

[0229] Input the enhanced current moment image information and enhanced images at other moments into the image feature extraction module for feature extraction to obtain corresponding image features, and output the image features to the feature map depth estimation module;

[0230] The image features and the depth map output by the feature map depth estimation module are input into the image feature mapping module to obtain the corresponding bird's-eye view features, and the bird's-eye view features are output to the target detection module.

[0231] Optionally, the feature map depth estimation module includes a first layer neural network and a second layer neural network. Before inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module, the method further includes:

[0232] Output the image features to the first layer of the neural network to obtain a first depth map output by the first layer of the neural network;

[0233] The first depth map and the feature data output by the first layer of the neural network are cascaded in the feature channel and input into the second layer of the neural network to obtain the second depth map output by the second layer of the neural network as the depth map output by the feature map depth estimation module.

[0234] Optionally, the target detection model further includes a temporal information fusion module, and the target detection result includes the speed of the target; inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module includes:

[0235] For the image information at each moment, the image features, the depth map output by the feature map depth estimation module, and the coordinate system related information at different moments are input into the image feature mapping module to obtain the corresponding bird's-eye view features;

[0236] The bird's-eye view features corresponding to each moment are input into the temporal information fusion module for fusion to obtain fused bird's-eye view features, and the fused bird's-eye view features are output to the target detection module.

[0237] Optionally, the target detection model further includes a feature decoding module, and outputting the bird's-eye view features to the target detection module includes:

[0238] Input the bird's-eye view features into the feature decoding module for decoding to obtain decoded bird's-eye view features;

[0239] The decoded bird's-eye view features are output to the target detection module.

[0240] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0241] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, terminal device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram and the combination of the process and / or box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for realizing the function specified in one process or multiple processes and / or one box or multiple boxes of the flow chart.

[0243] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0245] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0246] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0247] The above is a detailed introduction to a target detection model training method, a target detection method based on a target detection model, a target detection model training device, a target detection device based on a target detection model, an electronic device, and a non-volatile readable storage medium provided in this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A method for enhancing image information, characterized in that: include: Acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at a current moment and image information at other moments; Selecting candidate target samples from the target sample set; According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment; According to the coordinate system related information of the current moment and other moments, the alternative target sample is added to the image information of the other moments to obtain enhanced image information of the other moments, and the point cloud information corresponding to the alternative target sample is added to the point cloud information corresponding to the other moments, so that the position of the alternative target sample in the image information of the other moments relative to the position in the image information of the current moment meets the requirements of timing training.

2. The method according to claim 1, characterized in that The adding the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment comprises: According to the current moment image information in the image information sequence and the three-dimensional labels of the candidate target samples, as well as the coordinate system related information, the candidate target samples and the target samples in the current moment image information are added to the current moment image information in order of depth from far to near, so as to obtain enhanced current moment image information.

3. The method according to claim 1, characterized in that The adding the candidate target sample to the image information at other moments according to the coordinate system related information at the current moment and at other moments to obtain enhanced image information at other moments includes: According to the coordinate system related information of the current moment and other moments and the three-dimensional label of the candidate target sample, the candidate target sample and the target sample in the image information of other moments are added to the image information of other moments in order of depth from far to near to obtain enhanced image information of other moments; wherein the three-dimensional label includes speed information.

4. The method according to claim 1, characterized in that Before adding the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and adding the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment, the method further includes: Determine, according to the current moment image information and the three-dimensional label of the candidate target sample, and coordinate system related information, the current moment image information to which the candidate target sample should be added, and the camera that captures the current moment image information; Determine, according to the coordinate system related information of the current moment and other moments and the three-dimensional label of the candidate target sample, the image information of the other moments to which the candidate target sample should be added and the camera that acquires the image information of the other moments; If the camera that collects the image information at other moments and the camera that collects the image information at the current moment are the same camera, the candidate target samples are retained.

5. The method according to claim 1, characterized in that The selecting candidate target samples from the target sample set includes: Random sampling is performed from the target sample set to select the candidate target sample.

6. The method according to claim 1, characterized in that The coordinate system related information includes information describing the conversion relationship between the vehicle posture, the vehicle coordinate system, the camera coordinate system, and the radar coordinate system.

7. The method according to claim 3, characterized in that The speed information is used to determine whether the distance between the position of the candidate target sample in the image information at the current moment and the position in the image information at other moments is consistent with the speed information.

8. A method for training a target detection model based on image information enhancement, characterized in that: The method comprises: Acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at a current moment and image information at other moments; Selecting candidate target samples from the target sample set; According to the current moment image information, the candidate target sample is added to the current moment image information to obtain enhanced current moment image information, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the current moment; According to the coordinate system related information of the current moment and other moments, the candidate target sample is added to the image information of the other moments to obtain enhanced image information of the other moments, and the point cloud information corresponding to the candidate target sample is added to the point cloud information corresponding to the other moments, so that the position of the candidate target sample in the image information of the other moments relative to the position in the image information of the current moment meets the time series training requirements; The target detection model is trained using the enhanced current moment image information and enhanced other moment image information, target sample labeling data and the point cloud information; wherein the target detection result output by the target detection model includes the position information, length, width, height and speed of the target.

9. The method according to claim 8, characterized in that The target detection model includes an image feature extraction module, a feature map depth estimation module, an image feature mapping module and a target detection module. The use of the enhanced current moment image information and enhanced other moment image information, target sample labeling data and the point cloud information to train the target detection model includes: Inputting the enhanced current moment image information and enhanced images at other moments into the image feature extraction module for feature extraction to obtain corresponding image features, and outputting the image features to the feature map depth estimation module; The image features and the depth map output by the feature map depth estimation module are input into the image feature mapping module to obtain corresponding bird's-eye view features, and the bird's-eye view features are output to the target detection module.

10. The method according to claim 9, characterized in that The feature map depth estimation module includes a first layer of neural network and a second layer of neural network. Before inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module, the method further includes: Outputting the image features to the first layer of the neural network to obtain a first depth map output by the first layer of the neural network; The first depth map and the feature data output by the first layer of the neural network are cascaded in the feature channel and input into the second layer of the neural network to obtain a second depth map output by the second layer of the neural network as the depth map output by the feature map depth estimation module.

11. The method according to claim 9, characterized in thatThe target detection model further includes a temporal information fusion module, and the target detection result includes the speed of the target; the step of inputting the image features and the depth map output by the feature map depth estimation module into the image feature mapping module to obtain the corresponding bird's-eye view features, and outputting the bird's-eye view features to the target detection module includes: For the image information at each moment, the image features, the depth map output by the feature map depth estimation module and the coordinate system related information at different moments are respectively input into the image feature mapping module to obtain the corresponding bird's-eye view features; The bird's-eye view features corresponding to each moment are input into the time series information fusion module for fusion to obtain fused bird's-eye view features, and the fused bird's-eye view features are output to the target detection module.

12. The method according to claim 9, characterized in that The target detection model further includes a feature decoding module, and the outputting of the bird's-eye view features to the target detection module includes: Inputting the bird's-eye view feature into the feature decoding module for decoding to obtain a decoded bird's-eye view feature; The decoded bird's-eye view features are output to the target detection module.

13. The method according to claim 8, characterized in that The method further comprises: In the process of training the target detection model, the final loss function used is: L=λ depth L depth +λ cls L cls +λ reg L reg Among them, L depth is the depth estimation loss of image features, L cls is the target classification loss of image features, L reg is the target position, length, width, height, and speed estimation loss of the image features, λ depth For L depth The corresponding weight, λ cls For L cls The corresponding weight, λ reg For L reg The corresponding weight.

14. The method according to claim 9, characterized in that The true value of the depth map output by the feature map depth estimation module is determined by the point cloud information.

15. The method according to claim 9, characterized in that The target detection model further includes an image feature enhancement module, and the outputting of the image features to the feature map depth estimation module includes: The image features are input into the image feature enhancement module for feature enhancement to obtain enhanced image features; and the enhanced image features are output to the feature map depth estimation module.

16. The method according to claim 15, characterized in that The image feature enhancement module is used to learn the features of the image through upsampling to obtain the enhanced image features.

17. An image information enhancement device, characterized in that: include: An information acquisition module, used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments; A sample selection module is used to select candidate target samples from the target sample set; A first information enhancement module is used to add the candidate target sample to the image information at the current moment according to the image information at the current moment to obtain enhanced image information at the current moment, and add the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment; The second information enhancement module is used to add the alternative target sample to the image information of other moments according to the coordinate system related information of the current moment and other moments to obtain enhanced image information of other moments, and add the point cloud information corresponding to the alternative target sample to the point cloud information corresponding to the other moments, so that the position of the alternative target sample in the image information of other moments relative to the position in the image information of the current moment meets the requirements of time series training.

18. A target detection model training device based on image information enhancement, characterized in that: The device comprises: An information acquisition module, used to acquire image information sequences and corresponding point cloud information collected by multiple cameras; wherein the image information sequence includes image information at the current moment and image information at other moments; A sample selection module is used to select candidate target samples from the target sample set; A first sample adding module is used to add the candidate target sample to the image information at the current moment according to the image information at the current moment, so as to obtain enhanced image information at the current moment, and to add the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the current moment; A second sample adding module is used to add the candidate target sample to the image information at other moments according to the coordinate system related information of the current moment and other moments, so as to obtain enhanced image information at other moments, and add the point cloud information corresponding to the candidate target sample to the point cloud information corresponding to the other moments, so that the position of the candidate target sample in the image information at other moments relative to the position in the image information at the current moment meets the time series training requirements; The model training module is used to train the target detection model by using the enhanced current moment image information and enhanced other moment image information, target sample labeling data and the point cloud information; wherein the target detection result output by the target detection module includes the target's location information, length, width, height and speed.

19. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-16 when executing a program stored in a memory.

20. A non-volatile readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of method claims 1 to 16.

Images