Training method of target detection model, target detection method and device

Abstract

A training method of an object detection model, for any monocular camera, the internal parameter of the monocular camera is transformed into the extended internal parameter of another monocular camera, the image collected by the monocular camera is mapped into the image collected by the extended camera, and then the object detection model is trained. In the training stage, the internal parameter of the extended camera is used when generating 3D coordinates, so that the internal parameter of the camera is matched with the image, so that the model can be applied to the extended camera, and the generalization ability of the model is improved. A training device of an object detection device, an object detection method and device, a chip system, a program product, a storage medium and an electronic equipment are also disclosed.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a training method for an object detection model, an object detection method, and an apparatus. Background Technology

[0002] Object detection is a traditional task in computer vision. Unlike image recognition, object detection requires defining the location of the target object using a minimum bounding box. 3D object detection, in particular, requires providing the 3D bounding box of the target object. Taking autonomous driving as an example, 3D object detection obtains the 3D coordinates of the target object, then uses these coordinates to derive the 3D bounding box, and subsequently... Figure 1 The 3D frame is visualized on the image and bird's-eye view, as shown.

[0003] Several 3D object detection methods based on monocular cameras have been proposed in related technologies. These methods use object detection models to process images acquired by the monocular camera, obtaining the 3D vertices of the target object and thus a 3D bounding box. However, these methods cannot perform data augmentation through image-based geometric transformations, resulting in poor generalization ability. Summary of the Invention

[0004] This application provides a training method for an object detection model, an object detection method, and an apparatus to improve the generalization ability of a 3D object detection model based on a monocular camera.

[0005] Firstly, this application provides a method for training an object detection model, comprising:

[0006] Acquire the intrinsic parameters of at least one monocular camera and the images acquired by the at least one monocular camera;

[0007] The target detection model is trained N times based on the intrinsic parameters of the first camera and the images acquired by the first camera, wherein the first camera is any one of the at least one monocular camera, and N is an integer greater than 1.

[0008] The N training sessions include at least one first training session, which includes the following steps:

[0009] The intrinsic parameters of the first camera are transformed to obtain the extended intrinsic parameters used in the first training.

[0010] Geometric transformation is performed on the image acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample image used for the first training. The three-dimensional position information of the target object in the image acquired by the first camera is used as the labeled position information of the target object in the sample image.

[0011] The target detection model is used to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image.

[0012] Based on the extended intrinsic parameters, coordinate transformation is performed on the first two-dimensional position information and the first depth information to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters;

[0013] Based on the difference between the first three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0014] In this implementation, for any monocular camera, the intrinsic parameters of that monocular camera are transformed into the intrinsic parameters of another monocular camera (i.e., extended intrinsic parameters). The image acquired by the monocular camera is mapped to the image acquired by the extended camera, and then the object detection model is trained. During the training phase, the extended camera's intrinsic parameters are used for the image from the extended camera when generating 3D coordinates, so that the camera's intrinsic parameters are used for image matching. This allows the model to be applicable to extended cameras, improving the model's generalization ability. In other words, the same object detection model can be applied to cameras with different intrinsic parameters.

[0015] In some implementations, the N training sessions include at least one second training session, which includes the following steps:

[0016] The target detection model is used to detect the image captured by the first camera to obtain the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image captured by the first camera;

[0017] Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system;

[0018] Based on the difference between the second three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0019] In this implementation, the target detection model is trained on real cameras in the camera set, and the target detection model can be well applied to any camera in the camera set.

[0020] In some implementations, if the intrinsic parameters of the first camera are transformed in the S first training sessions out of the N training sessions, then the extended intrinsic parameters obtained from the transformation in each of the S first training sessions are different, where S is an integer less than or equal to N.

[0021] This approach allows the object detection model to be trained using more samples, thereby further improving the generalization ability of the object detection model.

[0022] In some implementations, transforming the intrinsic parameters of the first camera to obtain the extended intrinsic parameters used in the first training includes:

[0023] The intrinsic parameters of the first camera are randomly perturbed to obtain the extended intrinsic parameters used in the first training.

[0024] This approach ensures that the extended intrinsic parameters vary as much as possible around the intrinsic parameters of the first camera, thereby maximizing the accuracy of the object detection model's inference results and facilitating training convergence as quickly as possible. Random perturbations allow the intrinsic parameters of the first camera to fluctuate within a certain range, increasing the amount of data and improving the robustness of the object detection model.

[0025] In some implementations, the random perturbation of the intrinsic parameters of the first camera includes:

[0026] For the sub-parameters in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameters as the center;

[0027] Within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as an extended sub-parameter of the sub-parameter;

[0028] Replace the sub-parameter in the intrinsic parameters of the first camera with the extended sub-parameter of the sub-parameter.

[0029] This approach can generate a large number of extended intrinsic parameters, thereby increasing the number of training samples and improving the model's generalization ability.

[0030] In some implementations, the step of performing geometric transformations on the images acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample images used for the first training includes:

[0031] Based on the intrinsic parameters and distortion coefficients of the first camera, the image acquired by the first camera is distorted to obtain the distorted image.

[0032] The distortion-reduced image is processed based on the extended intrinsic parameters to obtain the sample image.

[0033] In this embodiment, distortion correction ensures that the image captured by the first camera is accurately mapped to the image space of the extended camera, thereby improving the accuracy of the model inference results.

[0034] Secondly, this application provides an object detection method, applied to the process of detecting target objects using an object detection model trained by the method described in any one of the first aspects, the method comprising:

[0035] Acquire the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera;

[0036] The image to be detected is inspected to obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected;

[0037] Based on the intrinsic parameters of the monocular camera, coordinate transformation is performed on the two-dimensional position information and the depth information to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

[0038] In this implementation, multiple cameras with different intrinsic parameters are used because their intrinsic parameters were used during model training. The model has adapted to its intrinsic parameters, so during inference, the inverse matrix of the intrinsic parameters can be used to left-multiply (u,v,Z') and obtain the correct 3D coordinates.

[0039] Thirdly, this application also provides a training device for an object detection model, comprising:

[0040] An information acquisition module is used to acquire the intrinsic parameters of at least one monocular camera and the images captured by the at least one monocular camera;

[0041] The training module is used to train the target detection model N times based on the intrinsic parameters of the first camera and the images acquired by the first camera, wherein the first camera is any one of the at least one monocular camera, and N is an integer greater than 1.

[0042] The N training sessions include at least one first training session, which includes the following steps:

[0043] The intrinsic parameters of the first camera are transformed to obtain the extended intrinsic parameters used in the first training.

[0044] Geometric transformation is performed on the image acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample image used for the first training. The three-dimensional position information of the target object in the image acquired by the first camera is used as the labeled position information of the target object in the sample image.

[0045] The target detection model is used to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image.

[0046] Based on the extended intrinsic parameters, coordinate transformation is performed on the first two-dimensional position information and the first depth information to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters;

[0047] Based on the difference between the first three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0048] In some implementations, the training module is further configured to perform at least one second training iteration during the N training iterations, the second training iteration comprising the following steps:

[0049] The target detection model is used to detect the image captured by the first camera to obtain the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image captured by the first camera;

[0050] Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system;

[0051] Based on the difference between the second three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0052] In some implementations, if the intrinsic parameters of the first camera are transformed in the S first training sessions out of the N training sessions, then the extended intrinsic parameters obtained from the transformation in each of the S first training sessions are different, where S is an integer less than or equal to N.

[0053] In some implementations, the training module is used to randomly perturb the intrinsic parameters of the first camera to obtain extended intrinsic parameters used in this training.

[0054] In some implementations, the random perturbation of the intrinsic parameters of the first camera is performed, and the training module is specifically used for:

[0055] For the sub-parameters in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameters as the center;

[0056] Within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as an extended sub-parameter of the sub-parameter;

[0057] Replace the sub-parameter in the intrinsic parameters of the first camera with the extended sub-parameter of the sub-parameter.

[0058] In some implementations, the training module is used to: perform distortion correction on the image acquired by the first camera based on the intrinsic parameters of the first camera and the distortion coefficients of the first camera, to obtain a distortion-corrected image;

[0059] The distortion-reduced image is processed based on the extended intrinsic parameters to obtain the sample image.

[0060] Fourthly, this application also provides a target detection apparatus, used in the process of detecting a target object using a target detection model obtained by the apparatus as described in any one of the third aspects, the apparatus comprising:

[0061] The image acquisition module is used to acquire the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera;

[0062] The two-dimensional information acquisition module is used to detect the image to be detected and obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected;

[0063] The three-dimensional information determination module is used to perform coordinate transformation on the two-dimensional position information and the depth information according to the intrinsic parameters of the monocular camera to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

[0064] Fifthly, this application provides a chip system comprising: a memory for storing a computer program; and a processor; wherein when the processor retrieves and runs the computer program from the memory, an electronic device equipped with the chip system performs the method as described in any one of the first and second aspects.

[0065] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method as described in any one of the first and second aspects.

[0066] In a seventh aspect, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first and second aspects.

[0067] Eighthly, this application also provides an electronic device, comprising:

[0068] Memory, used to store readable programs;

[0069] At least one processor is configured to call and run the readable program from the memory, such that the communication device implements the method as described in any one of the first and second aspects. Attached Figure Description

[0070] Figure 1A schematic diagram illustrating the visualization of 3D frames on images and bird's-eye views;

[0071] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0072] Figure 3 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0073] Figure 4 A schematic diagram of assembling a monocular camera in an autonomous vehicle;

[0074] Figure 5 A flowchart illustrating the training method for the target detection model provided in this application embodiment;

[0075] Figure 6 Another flowchart illustrating the training method for the target detection model provided in this application embodiment;

[0076] Figure 7 A schematic flowchart of the target detection method provided in the embodiments of this application;

[0077] Figure 8 This is a schematic diagram illustrating the use of the same target detection model by multiple monocular cameras in an embodiment of this application.

[0078] Figure 9 A schematic diagram of the structure of the training device for the target detection model provided in the embodiments of this application;

[0079] Figure 10 This is a schematic diagram of the target detection device provided in the embodiments of this application;

[0080] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0082] As mentioned in the background section, when performing 3D object detection based on images acquired by a monocular camera, the object detection model cannot use image-based geometric transformations, a common data augmentation method, for data augmentation. Consequently, it cannot be trained using data augmentation, resulting in poor generalization ability. This is because image-based geometric transformations affect the 2D-to-3D mapping relationship. The following explains how obtaining a 3D bounding box using an object detection model involves the following steps:

[0083] The first step involves inputting the image captured by the monocular camera into the object detection model. This model can be a depth prediction model based on 2D images. This model infers the 2D coordinates (u, v) of the target object in the image and predicts the corresponding depth Z. Typically, this model infers the 2D coordinates and depth of the center point of the 3D bounding box in the image. It may also infer the 2D coordinates and depth of a few other 3D bounding box vertices. The remaining 2D coordinates and depths of the uninferred 3D bounding box vertices can be calculated using the inferred 2D coordinates and depths of the 3D bounding box vertices and the geometry of obstacles. Since a 3D bounding box has 8 vertices plus the center point, this yields the 2D coordinates and depths of 9 key points.

[0084] The second step involves using the imaging principle of a monocular camera to analyze the detection results of the target detection model to obtain the 3D coordinates of the target object in the camera coordinate system. As shown in Equation 1, this describes the mapping relationship from 2D to 3D:

[0085]

[0086] In formula (1), (u, v) represent the 2D coordinates of the keypoint, Z represents the depth of the keypoint, and K represents the intrinsic parameters of the monocular camera, which can be obtained through camera calibration. Where fx, fy, cx, and cy are all sub-parameters of the intrinsic parameters, K -1 Let K be the inverse matrix, and (X, Y, Z) be the 3D coordinates of the pixel in the camera coordinate system of the monocular camera.

[0087] By taking the above two steps, the 3D coordinates of the target object can be obtained, and then the length, width, height and orientation angle of the target object can be deduced.

[0088] After performing geometric transformations on the image, such as translation, scaling, and rotation, the (u,v,Z) values ​​will differ from the K used in the second step. -1 The mismatch leads to the calculation of incorrect 3D coordinates.

[0089] Therefore, the object detection model for a monocular camera is bound to specific intrinsic parameters, making it impossible to apply a single model to multiple cameras with different intrinsic parameters. In other words, each monocular camera requires its own object detection model, and a single object detection model is only applicable to the monocular camera it is bound to. Consequently, simple image-based data augmentation of sample images is not suitable for the object detection model of a monocular camera, resulting in a lack of improvement in the generalization ability of the object detection model.

[0090] In view of this, in order to improve the generalization ability of the target detection model, this application provides a feasible data augmentation method to train the target detection model and improve the model's generalization ability.

[0091] In this embodiment, using the camera intrinsic parameter K of a monocular camera, geometric transformations can be performed on the image acquired by the monocular camera to achieve data augmentation. To maintain the correct 2D to 3D mapping, an intrinsic parameter K′ (hereinafter also referred to as the extended intrinsic parameter) can be constructed based on the intrinsic parameter K, and the original image is projected onto an image captured by an extended camera with intrinsic parameter K′. If K′ differs from K only in cx, the above transformation is equivalent to performing a left-right translation operation on the original image; if only cy differs, it is equivalent to performing a vertical translation operation on the original image; if only fx differs, it is equivalent to performing a horizontal scaling operation on the original image; if only fy differs, it is equivalent to performing a vertical scaling operation on the original image. Furthermore, for an image that has undergone geometric transformation using K′, any point (u,v,Z) on its plane is multiplied by the inverse matrix K′ of K′. -1 This allows you to obtain the correct 3D coordinates.

[0092] Based on this, in this embodiment, for any monocular camera, the intrinsic parameters of that monocular camera are transformed into the intrinsic parameters of another monocular camera (i.e., the extended intrinsic parameters of the extended camera), and the image acquired by the monocular camera is mapped to the image acquired by the extended camera. Then, the object detection model is trained. During the training phase, the extended camera's intrinsic parameters are used for the image of the extended camera when generating 3D coordinates, so that the camera's intrinsic parameters are used for image matching, thereby enabling the model to be applicable to the extended camera and improving the model's generalization ability. That is, the same object detection model can be applied to cameras with different intrinsic parameters.

[0093] In this embodiment, the object detection model can be trained on the server side or the terminal side. After training, the object detection model can be applied to the terminal. The terminal can be, for example, a vehicle, a mobile phone, a robot, or other device equipped with a monocular camera. Taking a vehicle as an example... Figure 2 The diagram illustrates an application scenario provided by an embodiment of this application. This application scenario includes a vehicle 101 and a target object 102. The vehicle 101 acquires an image containing the target object using its installed monocular camera, and this image can be input to the target detection model 103 within the vehicle 101 to detect the 3D position of the target object 102.

[0094] In another application scenario, object detection model 103 can also be applied to servers. For example... Figure 3 As shown, this application scenario includes a vehicle 101, a target object 102, and a server 104. The monocular camera of the vehicle 101 acquires an image containing the target object, and then the vehicle 101 sends the image to the server 104. The server 104 uses its built-in target detection model 103 to infer the 3D position of the target object from the image.

[0095] To facilitate a more intuitive understanding of the generalization ability of the target detection model in the embodiments of this application, the following is combined with... Figure 4 This needs to be explained. For example... Figure 4 As shown, the vehicle is equipped with four types of monocular cameras, totaling 16 cameras. The first type is a long-range camera (1 camera), the second type is a medium-range camera (4 cameras), the third type is a short-range camera (7 cameras), and the fourth type is a fisheye camera (4 cameras). Related technologies require training a separate object detection model for each monocular camera, resulting in 16 object detection models to be trained. However, using the method provided in this application, since the intrinsic parameters of cameras of the same type are not significantly different, one object detection model can be trained for each type of camera, ultimately requiring the training of 4 object detection models. Each object detection model is applicable to the same type of monocular camera, thereby improving the generalization ability of the object detection models.

[0096] like Figure 5 The diagram shown is a flowchart illustrating the training method of the target detection model in this embodiment of the application, including:

[0097] Step 501: Obtain the intrinsic parameters of at least one monocular camera and the images acquired by at least one monocular camera.

[0098] For example, such as Figure 4 As shown, a camera set can consist of monocular cameras of the same type. The cameras in the set are used to jointly train the same object detection model. In practice, it is not limited to the same camera type; as long as the intrinsic parameter difference is less than the difference threshold, monocular cameras of different types can also be used to construct a camera set for training the same object detection model. During training, any monocular camera in the set can be used as the first camera to train the object detection model.

[0099] Step 502: Based on the intrinsic parameters of the first camera and the images acquired by the first camera, train the target detection model N times, where N is an integer greater than 1.

[0100] Among them, the N training sessions include at least one first training session, which includes the following steps:

[0101] Step 5021: Transform the intrinsic parameters of the first camera to obtain the extended intrinsic parameters used for the first training.

[0102] In some implementations, if S out of N training iterations transform the intrinsic parameters of the first camera, then the extended intrinsic parameters obtained from each of the S transformations in the first training iterations are different, where S is an integer less than or equal to N. For example, the intrinsic parameters K of the first camera are transformed to obtain K1′, K2′, K3′...K′. mThere are m distinct extended intrinsic parameters. For any first camera, assuming it captures a total of p images and has m extended intrinsic parameters, this is equivalent to increasing the number of sample images by p*m. This increases the number of sample images used to train the object detection model, thereby further improving the generalization ability of the object detection model.

[0103] In some implementations, the intrinsic parameters of the first camera can be transformed in various ways, such as by translating any sub-parameter of the intrinsic parameters with equal or unequal steps, for example, the intrinsic parameters of the first camera. After shifting fx by step size d to obtain fx′, the extended intrinsic parameters are formed. Translation can be done in various ways, such as fx′ = fx + d, or fx′ = fx - d. Furthermore, adjustments can be made according to the desired direction of change. For example, if K′ differs from K only in cx, it's equivalent to a left-right translation of the original image; if only cy differs, it's equivalent to a vertical translation; if only fx differs, it's equivalent to a horizontal scaling; and if only fy differs, it's equivalent to a vertical scaling.

[0104] In this embodiment, to avoid a large discrepancy between the extended intrinsic parameters and the intrinsic parameters of the first camera, which could limit the inference ability of the object detection model and prevent it from converging, the extended intrinsic parameters are distributed around the intrinsic parameters of the first camera as much as possible. For example, the distance between them is within a threshold, to ensure that the accuracy of the object detection model's inference results meets the expected requirements and to achieve training convergence as quickly as possible. One possible implementation is to obtain the extended intrinsic parameters used in this training by randomly perturbing the intrinsic parameters of the first camera within a threshold. Randomly perturbing the intrinsic parameters of the first camera causes them to fluctuate within a certain range, increasing the amount of data and improving the robustness of the object detection model.

[0105] Compared to the traditional approach of using a fixed intrinsic parameter for a single object detection model, this application employs a variable intrinsic parameter data augmentation method. This allows the model to adapt to different intrinsic parameters during training, enabling it to adapt to multiple cameras with different intrinsic parameters during use. Therefore, it improves the model's generalization ability, allowing the same object detection model to be adapted to multiple cameras with different intrinsic parameters, enabling a single model to be used by multiple monocular cameras, thus reducing development costs.

[0106] In some possible implementations, in order to facilitate obtaining the extended intrinsic parameters through random perturbation, this application embodiment uses the original value (i.e., sub-parameter) in the intrinsic parameters K of the first camera as the center point, and uses a normal distribution to generate random values ​​to replace the original value in K based on a pre-set standard deviation. This can be implemented as follows: for the sub-parameter in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameter as the center; then, within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as the extended sub-parameter of the sub-parameter; then, the extended sub-parameter is used to replace the sub-parameter in the intrinsic parameters of the first camera to obtain the extended intrinsic parameters.

[0107] For example, the internal parameters of the first camera. Construct a normal distribution curve 1 for fx, and then obtain a point fx′ within a specified range centered at fx from the normal distribution curve 1. This constitutes the extended intrinsic parameter. Of course, in practice, the extended intrinsic parameters are not limited to differing from the intrinsic parameters of the first camera by only one sub-parameter. Multiple sub-parameters can differ. For example, in addition to constructing a normal distribution curve 1 for fx, a normal distribution curve 2 for cx can also be constructed. Then, a point cx′ close to cx can be obtained from normal distribution curve 2 and used to replace cx, thus obtaining the extended intrinsic parameters. Since the intrinsic parameters of the first camera have a total of 4 sub-parameters, one or any number of extended sub-parameters can be selected to construct extended intrinsic parameters.

[0108] Furthermore, it should be noted that the parameter differences between different extended intrinsic parameters can be differences within the same sub-parameter or differences between different sub-parameters. For example, for the normal distribution curve 1 of fx, multiple (potentially tens of thousands) different fx′ can be obtained, resulting in different extended intrinsic parameters. As another example, if not only fx′ differs, but other sub-parameters such as cx′ also differ, then different extended intrinsic parameters will be obtained.

[0109] The model training can run for hundreds of thousands of iterations, each using a different intrinsic parameter K′. This intrinsic parameter K′ either comes from the intrinsic parameters of the real camera or is generated by random perturbation based on the intrinsic parameters of the real camera. Therefore, the model uses a large amount of training samples, which improves its generalization ability.

[0110] In this application, different intrinsic parameter generation methods can produce different effects. For example, an autonomous vehicle has multiple cameras. These cameras can be calibrated to obtain their intrinsic parameter list, and then the intrinsic parameters in the list can be used cyclically during model training to train the model, achieving the effect of one model adapting to multiple cameras on the vehicle. As another example, a camera has default intrinsic parameter values ​​at the factory. However, due to manufacturing processes, its actual intrinsic parameters will differ from the default intrinsic parameters, but are roughly distributed around the default intrinsic parameters. Perturbed intrinsic parameters can be generated using a normal distribution centered on the default intrinsic parameters and applied to model training, allowing the model to adapt to all situations near its default intrinsic parameters, achieving the effect of one model adapting to one camera.

[0111] Step 5022: Perform geometric transformation on the image acquired by the first camera according to the intrinsic and extended intrinsic parameters of the first camera to obtain the first training sample image, and use the three-dimensional position information of the target object in the image acquired by the first camera as the annotation position information of the target object in the sample image.

[0112] Image geometric transformation, also known as image spatial transformation, maps coordinate positions in one image to new coordinate positions in another image without changing the pixel values. Geometric transformations in image processing are typically used to minimize geometric distortion caused by imaging angles, perspective, etc. Image geometric transformations can include translation, rotation, scaling, and orthographic projection. Geometric transformations can be implemented using spatial transformations and interpolation algorithms. The key to geometric transformations lies in the transformation parameters during the mapping process, which can be one or more of the following: translation components, scaling factors, and rotation angles. Generally, when performing geometric transformations on images acquired by a camera, camera intrinsic parameters do not need to be considered. In this embodiment, extended intrinsic parameters of an extended camera are used as transformation parameters to implement translation, scaling, and other operations on images acquired by a first camera. For example, if the extended intrinsic parameter K′ differs from the intrinsic parameter K of the first camera only in cx, it is equivalent to performing a left-right translation operation on the original image.

[0113] Furthermore, images captured by a monocular camera exhibit image distortion. Image distortion is introduced due to deviations in lens manufacturing precision and assembly processes, leading to distortion of the original image. Lens distortion is classified into two types: radial distortion and tangential distortion. Radial distortion is caused by the inherent characteristics of the lens itself (convex lens), resulting in light bending more away from the lens center than near it. Distortion is distributed along the lens radius and mainly includes barrel distortion and pincushion distortion. Tangential distortion is caused by the lens itself not being parallel to the camera sensor plane (imaging plane), often due to installation misalignment when the lens is attached to the lens module. Therefore, to ensure accurate mapping of the image captured by the first camera to the image space of the extended camera, in this embodiment, the image captured by the first camera is distorted based on the intrinsic parameters and distortion coefficients of the first camera to obtain a distorted image; then, the distorted image is processed based on the extended intrinsic parameters to obtain a sample image.

[0114] Step 5023: Use the target detection model to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image. In this embodiment, the first two-dimensional position information is, for example, the position coordinates of each key point of the target object in the sample image, such as the 2D coordinates (u, v) mentioned above. The first depth information is the depth corresponding to the 2D coordinates (u, v), such as Z in formula (1).

[0115] Step 5024: Perform coordinate transformation on the first two-dimensional position information and the first depth information according to the extended intrinsic parameters to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters. The first three-dimensional position information is the 3D coordinates of the key points of the target object calculated by formula (1).

[0116] Step 5025: Adjust the parameters of the target detection model based on the difference between the first three-dimensional position information and the labeled position information.

[0117] An object detection model contains multiple functions, each with its own parameters, which define the model's functionality. The purpose of training is to estimate and adjust the parameters of these functions based on the training dataset, allowing the model to learn the mapping from the input image to the desired result.

[0118] Taking autonomous vehicles as an example, the overall training process for extended intrinsic parameters can be summarized as follows: Figure 6 As shown: After acquiring the image, the 3D coordinates of the key points of the obstacles in the image need to be labeled to obtain the labeled position of the target object. The labeled position is saved in the label file. Therefore, the label file contains the true 3D coordinates (x, y, z) of the key points of the obstacles (i.e., the labeled position of the target object). During training:

[0119] First, in step 601, the image A, intrinsic parameter K, and distortion coefficient D acquired by the monocular camera are obtained.

[0120] In step 602, the intrinsic parameter K is transformed into the intrinsic parameter K′.

[0121] In step 603, image A is distorted using intrinsic parameter K and distortion coefficient D to obtain image A′. Then, in step 604, image A′ is geometrically transformed using extended intrinsic parameter K′ to obtain image B.

[0122] It should be noted that the execution order of steps 602 and 603 is not restricted.

[0123] In step 605, image B is input to the target detection model, which outputs the 2D coordinates and depth (u) of the key points of the obstacle. p ,v p Z p ).

[0124] In step 606, the inverse matrix K′ of the extended intrinsic parameter K′ is used. -1 Left multiplication (u p ,v p Z p The predicted 3D coordinates are obtained.

[0125] In step 607, the difference between the predicted 3D coordinates and the true 3D coordinates (x, y, z) is determined, and the parameters of the target detection model can be adjusted based on this difference.

[0126] During training, the training samples used in the same training batch belong to the same extended camera. Each training batch includes multiple training samples, and the total difference between the predicted 3D coordinates and the true 3D coordinates of all training samples in the same training batch is calculated to adjust the parameters of the object detection model.

[0127] In addition to using an extended camera to train the object detection model, in this embodiment, the aforementioned N training iterations also include at least one second training iteration. This second training uses the intrinsic parameters of the first camera to train the object detection model, and can be implemented as follows:

[0128] The target detection model is used to detect the image acquired by the first camera, and the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image acquired by the first camera are obtained.

[0129] Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system;

[0130] Based on the difference between the second-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0131] Therefore, it is ensured that the object detection model is trained on real cameras in the camera set, and the object detection model can be well applied to any camera in the camera set.

[0132] Based on the same inventive concept, embodiments of this application also provide a method for target detection using the above-described target detection model, such as... Figure 7 As shown, it includes the following steps:

[0133] Step 701: Obtain the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera.

[0134] Step 702: Detect the image to be detected to obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected.

[0135] Step 703: Perform coordinate transformation on the two-dimensional position information and depth information based on the intrinsic parameters of the monocular camera to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

[0136] Taking autonomous vehicles as an example, the camera set includes the intrinsic parameters C of multiple monocular cameras. These monocular camera intrinsic parameters C can be extended to generate multiple extended intrinsic parameters E. Therefore, the intrinsic parameter set F used during the training phase includes intrinsic parameters C and E. When deploying the object detection model on a real vehicle, the intrinsic parameters K” and distortion coefficients D of the monocular cameras used are first calibrated. Since variable intrinsic parameters are used during model training, the true intrinsic parameters K” used during inference are only a subset of the intrinsic parameter set F used during model training.

[0137] Image A is distorted using parameters K” and D to obtain image B. Image B is then fed into the object detection model for inference, yielding the 2D coordinates and depth (u, v, Z') of obstacle keypoints. The inverse matrix of K” is then multiplied by (u, v, Z') to obtain the 3D coordinates (X', Y', Z') of the obstacle keypoints. After predicting the 3D coordinates of multiple obstacle keypoints, the complete 3D bounding box information of the obstacle is inferred based on geometric relationships.

[0138] In summary, multiple cameras with different intrinsic parameters can be used to left-multiply (u,v,Z') during inference because their intrinsic parameters have been used during model training and the model has adapted to their intrinsic parameters. This allows the model to obtain the correct 3D coordinates by using the inverse matrix of the intrinsic parameters during inference.

[0139] like Figure 8As shown, the same camera type includes four monocular cameras, assuming they are Camera 1, Camera 2, Camera 3, and Camera 4 in that order. These four cameras share the same object detection model. When Camera 1 acquires an image and inputs it into the object detection model, the intrinsic parameter K1 and distortion coefficient D1 of Camera 1 are used to correct the distortion of the acquired image. The object detection model then obtains the intrinsic parameter K1 of Camera 1 and uses the inverse matrix of intrinsic parameter K1 to obtain the 3D coordinates of the obstacle. Similarly, for the image acquired by Camera 2, the intrinsic parameter K2 and distortion coefficient D2 of Camera 2 are used to correct the distortion of the acquired image. The object detection model then obtains the intrinsic parameter K2 of Camera 2 and uses the inverse matrix of intrinsic parameter K2 to obtain the 3D coordinates of the obstacle. The processing methods for Camera 3 and Camera 4 are similar and will not be elaborated here.

[0140] In summary, this application discloses an object detection model applicable to one or even multiple camera models. Due to manufacturing processes, even cameras of the same brand and model will have different intrinsic parameters. Training a corresponding model for each camera would obviously be too costly. By adopting the solution of this application, a single model can be trained and made applicable to one or even several camera models.

[0141] Based on the same inventive concept, this application also provides a training device 900 for an object detection model, such as... Figure 9 As shown, the training device 900 includes:

[0142] Information acquisition module 901 is used to acquire the intrinsic parameters of at least one monocular camera and the images captured by the at least one monocular camera;

[0143] Training module 902 is used to train the target detection model N times based on the intrinsic parameters of the first camera and the images acquired by the first camera, wherein the first camera is any one of the at least one monocular camera, and N is an integer greater than 1.

[0144] The N training sessions include at least one first training session, which includes the following steps:

[0145] The intrinsic parameters of the first camera are transformed to obtain the extended intrinsic parameters used in the first training.

[0146] Geometric transformation is performed on the image acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample image used for the first training. The three-dimensional position information of the target object in the image acquired by the first camera is used as the labeled position information of the target object in the sample image.

[0147] The target detection model is used to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image.

[0148] Based on the extended intrinsic parameters, coordinate transformation is performed on the first two-dimensional position information and the first depth information to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters;

[0149] Based on the difference between the first three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0150] In some implementations, the training module is further configured to perform at least one second training iteration during the N training iterations, the second training iteration comprising the following steps:

[0151] The target detection model is used to detect the image captured by the first camera to obtain the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image captured by the first camera;

[0152] Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system;

[0153] Based on the difference between the second three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

[0154] In some implementations, if the intrinsic parameters of the first camera are transformed in the S first training sessions out of the N training sessions, then the extended intrinsic parameters obtained from the transformation in each of the S first training sessions are different, where S is an integer less than or equal to N.

[0155] In some implementations, the training module is used to randomly perturb the intrinsic parameters of the first camera to obtain the extended intrinsic parameters used in the first training.

[0156] In some implementations, the random perturbation of the intrinsic parameters of the first camera is performed, and the training module is specifically used for:

[0157] For the sub-parameters in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameters as the center;

[0158] Within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as an extended sub-parameter of the sub-parameter;

[0159] Replace the sub-parameter in the intrinsic parameters of the first camera with the extended sub-parameter of the sub-parameter.

[0160] In some implementations, the training module is used for:

[0161] Based on the intrinsic parameters and distortion coefficients of the first camera, the image acquired by the first camera is distorted to obtain the distorted image.

[0162] The distortion-reduced image is processed based on the extended intrinsic parameters to obtain the sample image.

[0163] Based on the same inventive concept, this application also provides a target detection device 1000, which is applied to the process of detecting target objects using a target detection model obtained from a training device 900, such as... Figure 10 The target detection device 1000 includes:

[0164] The image acquisition module 1001 is used to acquire the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera;

[0165] The two-dimensional information acquisition module 1002 is used to detect the image to be detected and obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected.

[0166] The three-dimensional information determination module 1003 is used to perform coordinate transformation on the two-dimensional position information and the depth information according to the intrinsic parameters of the monocular camera to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

[0167] Based on the same inventive concept, this application provides a chip system, including: a memory for storing computer programs; a processor; when the processor calls and runs the computer program from the memory, it causes an electronic device equipped with the chip system to execute a training method or a target detection method of any of the target detection models described in this application.

[0168] Based on the same inventive concept, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the training method or object detection method of any of the object detection models described in this application.

[0169] Based on the same inventive concept, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform a training method or object detection method for any of the object detection models described in this application.

[0170] Based on the same inventive concept, embodiments of this application also provide an electronic device, which can have the following... Figure 11The structure shown indicates that the electronic device can be a computer device, or a chip or chip system that enables the computer device to implement the above methods.

[0171] like Figure 11 The illustrated electronic device 1100 may include at least one processor 1101, which is coupled to a memory to read and execute instructions in the memory to implement the steps of the training method or object detection method of the object detection model in the embodiments of this application. Optionally, the electronic device may also include a communication interface 1102 for supporting the electronic device in receiving or sending signaling or data. The communication interface 1102 in the electronic device can be used to interact with other electronic devices. The processor 1101 can be used to enable the electronic device to perform actions such as... Figure 5-7 The steps in any of the methods shown. Optionally, the electronic device may further include a memory 1103 storing computer instructions. The memory 1103 may be coupled to the processor 1101 and / or the communication interface 1102 to support the processor 1001 in calling the computer instructions in the memory 1103 to implement, for example... Figure 5-7 The steps in any of the methods shown in the present application; in addition, the memory 1103 may also be used to store data involved in the method embodiments of the present application, for example, to store data and instructions necessary for the communication interface 1002 to realize interaction, and / or to store configuration information such as WORM attributes necessary for the electronic device to execute the methods described in the embodiments of the present application.

[0172] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are invoked and executed by a computer, the computer can perform the methods involved in any possible design of the above-described method embodiments. In this application, the computer-readable storage medium is not limited; for example, it can be RAM (random-access memory), ROM (read-only memory), etc.

[0173] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, implementation can be entirely or partially in the form of computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data electronic device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0174] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a terminal device. Optionally, the processor and storage medium can also be housed in different components within the terminal device.

[0175] These computer instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A training method for an object detection model, characterized in that, include: Acquire the intrinsic parameters of at least one monocular camera and the images acquired by the at least one monocular camera; The target detection model is trained N times based on the intrinsic parameters of the first camera and the images acquired by the first camera, wherein the first camera is any one of the at least one monocular camera, and N is an integer greater than 1. The N training sessions include at least one first training session, which includes the following steps: The intrinsic parameters of the first camera are transformed to obtain the extended intrinsic parameters used in the first training. Geometric transformation is performed on the image acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample image used for the first training. The three-dimensional position information of the target object in the image acquired by the first camera is used as the labeled position information of the target object in the sample image. The target detection model is used to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image. Based on the extended intrinsic parameters, coordinate transformation is performed on the first two-dimensional position information and the first depth information to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters; Based on the difference between the first three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

2. The method according to claim 1, characterized in that, The N training sessions include at least one second training session, which includes the following steps: The target detection model is used to detect the image captured by the first camera to obtain the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image captured by the first camera; Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system; Based on the difference between the second three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

3. The method according to claim 1 or 2, characterized in that, If the intrinsic parameters of the first camera are transformed in the S first training sessions out of the N training sessions, then the extended intrinsic parameters obtained by the transformation in each of the S first training sessions are different, where S is an integer less than or equal to N.

4. The method according to claim 1 or 2, characterized in that, The transformation of the intrinsic parameters of the first camera to obtain the extended intrinsic parameters used in the first training includes: The intrinsic parameters of the first camera are randomly perturbed to obtain the extended intrinsic parameters used in the first training.

5. The method according to claim 4, characterized in that, The random perturbation of the intrinsic parameters of the first camera includes: For the sub-parameters in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameters as the center; Within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as an extended sub-parameter of the sub-parameter; Replace the sub-parameter in the intrinsic parameters of the first camera with the extended sub-parameter of the sub-parameter.

6. The method according to claim 1 or 2, characterized in that, The step of performing geometric transformations on the images acquired by the first camera based on the intrinsic parameters and the extended intrinsic parameters of the first camera to obtain the sample images used for the first training includes: Based on the intrinsic parameters and distortion coefficients of the first camera, the image acquired by the first camera is deformed to obtain the deformed image. The distortion-reduced image is processed based on the extended intrinsic parameters to obtain the sample image.

7. A target detection method, characterized in that, The process of using a target detection model trained by the method described in any one of claims 1-6 to detect target objects, the method comprising: Acquire the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera; The image to be detected is inspected to obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected; Based on the intrinsic parameters of the monocular camera, coordinate transformation is performed on the two-dimensional position information and the depth information to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

8. A training device for an object detection model, characterized in that, include: An information acquisition module is used to acquire the intrinsic parameters of at least one monocular camera and the images captured by the at least one monocular camera; The training module is used to train the target detection model N times based on the intrinsic parameters of the first camera and the images acquired by the first camera, wherein the first camera is any one of the at least one monocular camera, and N is an integer greater than 1. The N training sessions include at least one first training session, which includes the following steps: The intrinsic parameters of the first camera are transformed to obtain the extended intrinsic parameters used in the first training. Geometric transformation is performed on the image acquired by the first camera based on the intrinsic parameters of the first camera and the extended intrinsic parameters to obtain the sample image used for the first training. The three-dimensional position information of the target object in the image acquired by the first camera is used as the labeled position information of the target object in the sample image. The target detection model is used to detect the sample image to obtain the first two-dimensional position information and the first depth information of the target object in the coordinate system of the sample image. Based on the extended intrinsic parameters, coordinate transformation is performed on the first two-dimensional position information and the first depth information to obtain the first three-dimensional position information of the target object in the camera coordinate system corresponding to the extended intrinsic parameters; Based on the difference between the first three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

9. The apparatus according to claim 8, characterized in that, The training module is further configured to perform at least one second training iteration during the N training iterations, the second training iteration comprising the following steps: The target detection model is used to detect the image captured by the first camera to obtain the second two-dimensional position information and the second depth information of the target object in the image coordinate system of the image captured by the first camera; Based on the intrinsic parameters of the first camera, coordinate transformation is performed on the second two-dimensional position information and the second depth information to obtain the second three-dimensional position information of the target object in the first camera coordinate system; Based on the difference between the second three-dimensional location information and the labeled location information, the parameters of the target detection model are adjusted.

10. The apparatus according to claim 8 or 9, characterized in that, If the intrinsic parameters of the first camera are transformed in the S first training sessions out of the N training sessions, then the extended intrinsic parameters obtained by the transformation in each of the S first training sessions are different, where S is an integer less than or equal to N.

11. The apparatus according to claim 8 or 9, characterized in that, The training module is used to randomly perturb the intrinsic parameters of the first camera to obtain the extended intrinsic parameters used in the first training.

12. The apparatus according to claim 11, characterized in that, The training module is specifically used to perform random perturbation of the intrinsic parameters of the first camera. For the sub-parameters in the intrinsic parameters of the first camera, a normal distribution curve is constructed with the sub-parameters as the center; Within a specified range centered on the sub-parameter, a point is obtained from the normal distribution curve, and the obtained point is used as an extended sub-parameter of the sub-parameter; Replace the sub-parameter in the intrinsic parameters of the first camera with the extended sub-parameter of the sub-parameter.

13. The apparatus according to claim 8 or 9, characterized in that, The training module is used for: Based on the intrinsic parameters and distortion coefficients of the first camera, the image acquired by the first camera is deformed to obtain the deformed image. The distortion-reduced image is processed based on the extended intrinsic parameters to obtain the sample image.

14. A target detection device, characterized in that, The process of detecting a target object using a target detection model obtained by the apparatus as described in any one of claims 8-13, wherein the apparatus comprises: The image acquisition module is used to acquire the image to be detected captured by the monocular camera and the intrinsic parameters of the monocular camera; The two-dimensional information acquisition module is used to detect the image to be detected and obtain the two-dimensional position information and depth information of the target object in the coordinate system of the image to be detected; The three-dimensional information determination module is used to perform coordinate transformation on the two-dimensional position information and the depth information according to the intrinsic parameters of the monocular camera to obtain the three-dimensional position information of the target object in the monocular camera coordinate system.

15. A chip system, characterized in that, include: Memory, used to store computer programs; processor; When the processor retrieves and runs a computer program from memory, it causes an electronic device equipped with the chip system to perform the method as described in any one of claims 1-7.

16. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

18. An electronic device, characterized in that, include: Memory, used to store readable programs; At least one processor is configured to call and run the readable program from the memory, such that the electronic device implements the method as described in any one of claims 1 to 7.

Images