Parking area determination method and device, equipment and storage medium

Abstract

The invention provides a parking area determination method and device, equipment and a storage medium, and the method comprises the steps: extracting first image features of at least two original images, the first image features comprising position features and height features under a first three-dimensional view coordinate system; for any direction, converting the first image feature corresponding to the direction to obtain a second image feature; performing feature fusion on the second image features in each direction in the panoramic direction of the vehicle to obtain panoramic image features of the vehicle; and extracting position features and height features of obstacles in the surrounding area of the vehicle from the panoramic image features. According to the technical scheme of the embodiment of the invention, the first image feature including the position feature and the height feature is extracted from the original image, and the panoramic image feature is obtained by processing the first image feature, so that the suspended obstacle can be identified from the panoramic image feature, the accuracy of the determined parking area is improved, and the parking efficiency is improved. And the parking effect is optimized.

Description

Technical Field

[0001] The present application belongs to the technical field of automatic parking, and specifically relates to a parking area determination method, device, equipment and storage medium. Background Art

[0002] With the development of smart cars, automatic parking has become a must-have configuration for many models. However, the current automatic parking technology only supports simple parking area determination based on parking space markings and ground obstacles. It cannot accurately identify suspended obstacles in the space, which may lead to collision risks during parking, resulting in poor parking results. Summary of the invention

[0003] The present application proposes a parking area determination method, device, equipment and storage medium. By extracting image features including position features and height features from the original image, and processing the image features to obtain a panoramic image feature of the vehicle, it is possible to identify suspended obstacles from the panoramic image features, which is beneficial to improving the accuracy of the determined parking area and optimizing the parking effect.

[0004] The first embodiment of the present application provides a parking area determination method, including:

[0005] Extracting first image features of at least two original images respectively, wherein the at least two original images include original images in each direction in the panoramic direction of the vehicle, the first image feature of any original image is a feature in a first three-dimensional viewing angle coordinate system, and the first image feature of any original image includes a position feature and a height feature of an image content in the original image;

[0006] For any direction of the panoramic direction of the vehicle, convert the first image feature corresponding to the direction, and the converted feature is the second image feature of the direction, and the second image feature is a feature in a second three-dimensional viewing angle coordinate system, and the second three-dimensional viewing angle coordinate system refers to the coordinate system of the panoramic direction of the vehicle;

[0007] Performing feature fusion on the second image features in each direction of the panoramic direction of the vehicle, wherein the fused features are the panoramic image features of the vehicle;

[0008] The position features and height features of obstacles in the area around the vehicle are extracted from the panoramic image features, and the drivable area determined by the position features and height features of the obstacles is the parking area of ​​the vehicle.

[0009] In some embodiments of the present application, for any direction of the panoramic direction of the vehicle, converting the first image feature corresponding to the direction, and the converted feature is the second image feature of the direction, including:

[0010] Determining a conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system;

[0011] For any direction of the panoramic direction of the vehicle, calculate the distortion parameter of each pixel feature in the first image feature corresponding to the direction, wherein the distortion parameter of any pixel feature is the distortion parameter of the pixel feature relative to a reference coordinate system, and the reference coordinate system is a rectangular three-dimensional coordinate system matching the first three-dimensional viewing angle coordinate system;

[0012] Determining, according to the distortion parameters of the respective pixel features, reference pixel features of the corresponding pixel features in the reference coordinate system, so as to obtain orthogonal image features in the reference coordinate system;

[0013] According to the conversion relationship, the orthodontic image features are converted into image features in the second three-dimensional viewing angle coordinate system to obtain second image features in the direction.

[0014] In some embodiments of the present application, calculating the distortion parameter of each pixel feature in the first image feature corresponding to the direction includes:

[0015] For any pixel feature, calculating the radius and the incident angle of the pixel feature in the reference coordinate system;

[0016] Calculating the distorted incident angle according to the incident angle in the reference coordinate system;

[0017] The distortion arc length of the pixel feature is calculated according to the distortion incident angle, and the distortion arc length is a distortion parameter of the pixel feature.

[0018] In some embodiments of the present application, the conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system is used to represent at least one of the following:

[0019] Any pixel coordinate in the second three-dimensional viewing angle coordinate system is converted to a rotation angle and a translation position in the first three-dimensional viewing angle coordinate system.

[0020] In some embodiments of the present application, the position characteristics of the obstacles in the area surrounding the vehicle include: a probability value of the position being an obstacle;

[0021] The height characteristics of the obstacles in the area around the vehicle include: a first height value and a second height value of the obstacle, the first height value refers to the distance between the lowest point of the obstacle and the ground, and the second height value refers to the distance between the highest point of the obstacle and the ground.

[0022] In some embodiments of the present application, the step of fusing the second image features in each direction in the panoramic direction of the vehicle includes:

[0023] For any two adjacent directions of the panoramic direction of the vehicle, pixel features of the same pixels in the overlapping area of ​​the second image features in the two directions are added.

[0024] In some embodiments of the present application, respectively extracting first image features of at least two original images includes:

[0025] For any original image, extract features of the original image to obtain at least two image sub-features, where the at least two image sub-features represent features of the original image at at least one size;

[0026] The at least two image sub-features are subjected to feature fusion, and the fused feature is the first image feature of the original image.

[0027] In some embodiments of the present application, the original image is a fisheye image.

[0028] In some embodiments of the present application, the method further includes: training the model to be trained to obtain a parking area determination model, wherein the parking area determination model is used to execute the parking area determination method:

[0029] The training method comprises:

[0030] Calling the model to be trained to identify a sample image to obtain a position prediction result and a height prediction result, wherein the sample image includes a plurality of groups of annotated images under a first three-dimensional perspective;

[0031] Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the type prediction result and the labeled height feature, and the cross entropy loss function is used to characterize the loss between the position prediction result and the labeled position feature;

[0032] Determine whether the smoothing loss function and the cross entropy loss function converge;

[0033] If both the smoothing loss function and the cross entropy loss function converge, using the to-be-trained model as the parking area determination model;

[0034] If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the model with adjusted parameters as a new model to be trained, and again perform the operation of calling the model to be trained to identify the sample image.

[0035] An embodiment of a second aspect of the present application provides a parking area determination device, including:

[0036] A first feature extraction module is used to extract first image features of at least two original images respectively, wherein the at least two original images include original images in various directions in the panoramic direction of the vehicle, the first image feature of any original image is a feature in a first three-dimensional viewing angle coordinate system, and the first image feature of any original image includes a position feature and a height feature of the image content in the original image;

[0037] a feature conversion module, for converting, for any direction of the panoramic direction of the vehicle, a first image feature corresponding to the direction, wherein the converted feature is a second image feature of the direction, wherein the second image feature is a feature in a second three-dimensional viewing angle coordinate system, wherein the second three-dimensional viewing angle coordinate system refers to a coordinate system of the panoramic direction of the vehicle;

[0038] A feature fusion module, used for fusing the second image features in each direction in the panoramic direction of the vehicle, and the fused features are the panoramic image features of the vehicle;

[0039] The second feature extraction module is used to extract the position features and height features of obstacles in the area around the vehicle from the panoramic image features, and the drivable area determined by the position features and height features of the obstacles is the parking area of ​​the vehicle.

[0040] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0041] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0042] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0043] In an embodiment of the present application, first, the first image features of at least two original images are extracted respectively. The original image here may refer to the original image in each direction in the panoramic direction of the vehicle, and the original image includes the position and height of the image content. Correspondingly, the first image feature includes the position feature and the height feature. In other words, the scheme of the embodiment of the present application supports the extraction of the height feature of the image content in space. In this way, through the combination of the height feature and the position feature, it can provide support for the identification of suspended obstacles. In this scheme, the first image feature is a feature in the first three-dimensional perspective coordinate system, and the first three-dimensional perspective coordinate system is not a three-dimensional perspective coordinate system under the panoramic perspective of the vehicle. Further, the first image feature in each direction of the panoramic direction of the vehicle is converted into a second image feature in the corresponding direction, and the second image feature is a three-dimensional perspective coordinate system under the panoramic perspective of the vehicle. The second image feature is obtained by converting the first image feature, and based on this, the second image feature also includes a position feature and a height feature. Then, after the second image features in all directions are fused into the panoramic image features of the vehicle, the position features and height features of the obstacles in the area around the vehicle can be extracted from the panoramic image features, and the combination of the position features and height features of the obstacles can characterize the features of the obstacles in space, that is, the position features and height features of the obstacles can characterize the suspended obstacles, and then the parking area determined according to the position features and height features of the obstacles is relatively accurate. It can be seen that in the embodiment of the present application, by extracting the first image features including the position features and the height features from the original image, and processing the first image features to obtain the panoramic image features, it is possible to identify the suspended obstacles from the panoramic image features, which is conducive to improving the accuracy of the determined parking area and optimizing the parking effect.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0046] In the attached picture:

[0047] Figure 1 A method flow chart of a parking area determination method provided by an embodiment of the present application is shown;

[0048] Figure 2A A schematic diagram of data flow based on an algorithm network provided in an embodiment of the present application is shown;

[0049] Figure 2B A flow chart of a method for converting image features in a camera coordinate system into image features in a vehicle coordinate system provided by an embodiment of the present application is shown;

[0050] Figure 3 A schematic diagram of the structure of a parking area determination device provided by an embodiment of the present application is shown;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0052] Figure 5 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0054] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0055] The embodiments of the present application relate to an automatic parking scenario. In the automatic parking scenario, before parking, the vehicle can determine the drivable area (free space) (i.e., the parking area) for parking by identifying obstacles, and then perform parking planning.

[0056] In some typical parking area determination schemes, the automatic parking system can identify the position of obstacles on the image of the Around View Monitor (AVM) system. The boundary formed by the contour lines of the obstacle position can be considered as the free space boundary, and then the area within the free space boundary is determined as the parking area. However, the contour lines of the obstacle position identified by the existing schemes are limited to the ground position of the obstacle, and the height of the obstacle cannot be identified. Based on this, the existing parking area determination method cannot identify suspended obstacles, and there may be a risk of collision during the parking process, resulting in poor parking results.

[0057] In view of this, the technical solution provided by the embodiment of the present application, after obtaining the original images in each direction in the panoramic direction of the vehicle, no longer splices the original images in each direction to obtain a panoramic image, but extracts the first image features of each original image separately. Since the original image includes the position and height of the image content, the first image features include position features and height features. Furthermore, according to the conversion relationship between the first three-dimensional perspective coordinate system corresponding to the original image and the three-dimensional perspective coordinate system under the panoramic perspective of the vehicle, each first image feature is converted to obtain a second image feature, and the second image feature is a three-dimensional perspective coordinate system under the panoramic perspective of the vehicle, and also includes position features and height features. Then, the panoramic image features obtained by fusion of the second image features can extract the position features and height features of obstacles in the area around the vehicle, and the combination of the position features and height features of the obstacles can characterize the features of the obstacles in space, that is, identifying suspended obstacles, which is conducive to improving the accuracy of the determined parking area and optimizing the parking effect.

[0058] The execution subject of this technical solution can be any device that supports the parking area determination function, including vehicles, ships, aircraft or robots. Such devices can support the parking area determination of the embodiment of this application through an automatic parking system. The automatic parking system can deploy an algorithm network for parking area determination.

[0059] A parking area determination method, apparatus, device, and storage medium proposed according to an embodiment of the present application are described below in conjunction with the accompanying drawings.

[0060] See also Figure 1 , Figure 1 A flow chart of a parking area determination method provided by an embodiment of the present application is shown, and the method specifically comprises the following steps:

[0061] Step S101: extract first image features of at least two original images respectively.

[0062] Among them, at least two original images may include original images in each direction of the panoramic direction of the vehicle. For example, the original image is a fisheye image (i.e., an image taken by a fisheye camera), and at least two original images may be fisheye images in four directions of the vehicle. The four directions are, for example, the front, rear, left, and right directions of the vehicle's surround view, and the fisheye images in the four directions may cover the panoramic view of the vehicle's surround view.

[0063] The first image feature of any original image is a feature in a first three-dimensional viewing angle (e.g., fisheye camera) coordinate system. The original image includes the position and height of the image content, so the first image feature of any original image may include the position feature and height feature of the image content in the original image.

[0064] In some embodiments, for each original image, the automatic parking system may pre-process the original image to obtain image data of the original image, and then perform computer vision operations on the image data to extract image features from the image data. Pre-processing the original image may include performing noise reduction, filtering, normalization, and other processing on the pixels of the original image to obtain red, green, and blue (RGB) three-channel data of the pixels of the original image. For example, the RGB three-channel data of the original image may include the RGB mean of the pixels in the original image and the RGB variance of the pixels in the original image.

[0065] Further, feature extraction is performed on the image data to obtain a first image feature representing a basic feature of the original image. Exemplarily, a DLANetDD3D network may be called to perform feature extraction on the image data.

[0066] In some embodiments, the image data may be operated once to obtain an image feature of an original image of a certain size, and the image feature of the certain size is the first image feature. In other embodiments, the image data may be operated at least twice, and each operation may obtain an image sub-feature of an original image of a certain size, and the sizes of the original images corresponding to the at least two operations are different. The at least two image sub-features constitute the first image feature.

[0067] In this way, if the image feature includes image sub-features of two sizes, the image feature can represent the features of obstacles of different sizes. By adopting this implementation method, original images of different sizes or features of obstacles of different sizes in the original image can be extracted, thereby improving the accuracy of obstacle recognition.

[0068] It should be understood that the size of the original image described in the embodiment of the present application may refer to the width (pixel (pixel, P))*height (P) of the original image.

[0069] With this implementation, since the original image includes the position and height of the image content, after acquiring the original image, the original images in various directions are no longer stitched together to obtain a panoramic image. Instead, the first image features of each original image are extracted separately. In this way, the first image features including the position features and the height features can be obtained, providing data support for identifying suspended obstacles.

[0070] Step S102: for any direction of the panoramic direction of the vehicle, convert the first image feature corresponding to the direction, and the converted feature is the second image feature of the direction.

[0071] The second three-dimensional perspective coordinate system refers to a coordinate system in the panoramic direction of the vehicle, that is, a three-dimensional perspective coordinate system of a bird's eye view (BEV) of the area around the vehicle. The second image feature is a feature in the second three-dimensional perspective coordinate system.

[0072] It should be pointed out that the second three-dimensional perspective coordinate system, as a BEV coordinate system, can be a relatively standard three-dimensional perspective coordinate system, while the first three-dimensional perspective coordinate system (such as a fisheye camera coordinate system) has an angle rotation and a position translation compared to the second three-dimensional perspective coordinate system. Accordingly, the pixels in the first three-dimensional perspective coordinate system are distorted compared to those in the second three-dimensional perspective coordinate system.

[0073] Based on this, the conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system can be determined first. Then, for any direction of the panoramic direction of the vehicle, the distortion parameters of each pixel feature in the first image feature corresponding to the direction are calculated, and according to the distortion parameters of each pixel feature, the reference pixel feature of the corresponding pixel feature in the reference coordinate system is determined to obtain the orthogonal image feature in the reference coordinate system, and according to the conversion relationship, the orthogonal image feature is converted into the image feature in the second three-dimensional viewing angle coordinate system to obtain the second image feature in the direction.

[0074] In some embodiments, for any direction of the panoramic direction of the vehicle, a rectangular three-dimensional coordinate system can be used as the reference coordinate system of the original image, that is, a Cartesian coordinate system including three axes of the x-axis, the y-axis, and the z-axis can be used as the reference coordinate system. Furthermore, for any pixel feature, the radius (i.e., the arc length from the origin) and the parameters such as the incident angle of the pixel feature in the reference coordinate system can be first calculated, that is, the variable diameter and the incident angle of the pixel feature in the non-distorted state are calculated. Among them, in the non-distorted state, the incident angle of the pixel feature should be equal to the exit angle, while in the first three-dimensional viewing angle coordinate system, the incident angle of the pixel feature is not equal to the exit angle. Based on this, the distorted incident angle can be calculated according to the incident angle in the reference coordinate system, and the distorted arc length of the pixel feature can be calculated according to the distorted incident angle, and the distorted arc length can characterize the distortion parameter of the pixel feature.

[0075] Exemplarily, the second three-dimensional perspective coordinate system is a relatively standard three-dimensional perspective coordinate system, and the distortion degree of the first three-dimensional perspective coordinate system is affected by the parameters of the fisheye camera. Then, the coordinates of each pixel in the second three-dimensional perspective coordinate system can be used as a reference to calculate the rotation angle and translation position of the pixel converted to the first three-dimensional perspective coordinate system, thereby determining the conversion relationship between the second three-dimensional perspective coordinate system and the first three-dimensional perspective coordinate system.

[0076] The conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system satisfies, for example: P=RPw +T, where P refers to the pixel coordinates of the first three-dimensional viewing angle coordinate system, P w refers to the pixel coordinates of the second three-dimensional viewing angle coordinate system, R refers to the angle rotation matrix, and T refers to the position translation matrix.

[0077] Furthermore, the position of each orthogonally rectified reference pixel in the reference coordinate system corresponds to a position in the second three-dimensional perspective coordinate system of the direction, and the position in the reference coordinate system and the position in the second three-dimensional perspective coordinate system satisfy the above-mentioned conversion relationship. Based on this, for the reference image feature corresponding to the original image in any direction, the pixel of the reference image feature in the direction can be projected to the corresponding image in the second three-dimensional perspective coordinate system of the direction according to the above-mentioned conversion relationship, and the projected second three-dimensional perspective coordinate system image feature is the second image feature in the direction.

[0078] First, the pixel features corresponding to the multiple position points in the image features are obtained, that is, the image features are sampled according to the multiple pre-deployed position points to obtain the pixel features of the multiple position points. Combined with the aforementioned relationship between the multiple position points and the polar coordinates, it can be known that the multiple position points are in a one-to-one correspondence with the positions indicated by the eigenvalues ​​in the coordinate mapping relationship matrix, so the image features of each position point correspond to the corresponding eigenvalues ​​in the coordinate mapping relationship matrix. To this end, the sorting of the feature points in the pixel features can be deployed according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix. The sorted pixel features can include the feature points, as well as the positional features of the feature points in the first coordinate system and the second coordinate system, and the sorted pixel features can be considered as the coordinate features.

[0079] Step S103, performing feature fusion on the second image features in each direction in the panoramic direction of the vehicle, and the fused features are the panoramic image features of the vehicle.

[0080] The panoramic image feature of the vehicle may be a BEV panoramic image feature of the area surrounding the vehicle, and the BEV panoramic image feature may include pixel features of images of four directions of the vehicle's surround view.

[0081] In some embodiments, the pixel features of the same pixels in the overlapping area of ​​the second image features in any two adjacent directions in the panoramic direction of the vehicle may be added, and the independent pixels in each direction may be retained to obtain the BEV image features of the vehicle.

[0082] The BEV image features of the vehicle obtained by this implementation are obtained by fusing multiple second image features, each of which is converted from a corresponding first image feature, and each first image feature contains the position feature and height feature of the image content. Correspondingly, the BEV image features of the vehicle can include the position feature and height feature of the obstacle, and the position feature and height feature can characterize the features of the suspended obstacle. In this way, the BEV image features of the vehicle of this application can identify the suspended obstacle.

[0083] Step S104: extracting position features and height features of obstacles in the area surrounding the vehicle from the panoramic image features.

[0084] Among them, the suspended obstacles can be determined according to the location characteristics and height characteristics of the obstacles.

[0085] Exemplarily, the position feature of the obstacle in the area around the vehicle includes: a probability value of the position being an obstacle, which can also be represented as a probability value of whether the position is occupied. In some embodiments, the larger the probability value, the greater the possibility that the position is occupied; conversely, the smaller the probability value, the smaller the possibility that the position is occupied.

[0086] The height characteristics of obstacles in the area around the vehicle may include: a first height value and a second height value of the obstacle, wherein the first height value refers to the distance between the lowest point of the obstacle and the ground, and the second height value refers to the distance between the highest point of the obstacle and the ground. In other words, the height characteristics of the obstacle are used to characterize the position characteristics of the obstacle in space. In this way, suspended obstacles can be identified.

[0087] In one implementation scenario, after identifying obstacles and suspended obstacles in the panoramic image features, the outline composed of various obstacles can be determined in the panoramic image. The outline can be considered as the boundary of the vehicle's drivable area. Then, the area enclosed by the outline composed of the obstacles can be determined as the parking area of ​​the vehicle.

[0088] In an embodiment of the present application, first, the first image features of at least two original images are extracted respectively. The original image here may refer to the original image in each direction in the panoramic direction of the vehicle, and the original image includes the position and height of the image content. Correspondingly, the first image feature includes the position feature and the height feature. In other words, the scheme of the embodiment of the present application supports the extraction of the height feature of the image content in space. In this way, through the combination of the height feature and the position feature, it can provide support for the identification of suspended obstacles. In this scheme, the first image feature is a feature in the first three-dimensional perspective coordinate system, and the first three-dimensional perspective coordinate system is not a three-dimensional perspective coordinate system under the panoramic perspective of the vehicle. Further, the first image feature in each direction of the panoramic direction of the vehicle is converted into a second image feature in the corresponding direction, and the second image feature is a three-dimensional perspective coordinate system under the panoramic perspective of the vehicle. The second image feature is obtained by converting the first image feature, and based on this, the second image feature also includes a position feature and a height feature. Then, after the second image features in all directions are fused into the panoramic image features of the vehicle, the position features and height features of the obstacles in the area around the vehicle can be extracted from the panoramic image features, and the combination of the position features and height features of the obstacles can characterize the features of the obstacles in space, that is, the position features and height features of the obstacles can characterize the suspended obstacles, and then the parking area determined according to the position features and height features of the obstacles is relatively accurate. It can be seen that in the embodiment of the present application, by extracting the first image features including the position features and the height features from the original image, and processing the first image features to obtain the panoramic image features, it is possible to identify the suspended obstacles from the panoramic image features, which is conducive to improving the accuracy of the determined parking area and optimizing the parking effect.

[0089] Furthermore, Figure 1 Some or all of the steps of the illustrated method embodiment may be executed by a pre-deployed algorithm network.

[0090] An exemplary algorithm network is as follows Figure 2A As shown, according to the data transmission direction, the algorithm network may include a backbone network, a feature processing network and a prediction network, wherein the feature processing network may be the neck network of the algorithm network, and the prediction network may be the head network of the algorithm network. In some embodiments, the backbone network may be used to perform the operation processing of the above-mentioned embodiment related to S101, the feature processing network may be used to perform the operation processing of the above-mentioned embodiment related to S102 and S103, and the prediction network may be used to perform the operation processing of the above-mentioned embodiment related to S104.

[0091] It should be understood that Figure 2AThe illustrated structure does not constitute a specific limitation on the algorithm network of the present application. In other embodiments of the present application, the algorithm network may include more or fewer component networks than shown, or combine certain networks, or split certain networks, or arrange different networks. It will be apparent to those skilled in the art that many modifications and variations are possible based on the above teachings.

[0092] The backbone network can be implemented as any algorithm model that supports extracting the first image feature from the original image, including but not limited to DLANetDD3D network, etc. The feature processing network can be implemented as any algorithm network that supports conversion processing between three-dimensional coordinate systems, pixel projection, etc., or a combination of algorithm networks, for example, the feature processing network can be implemented as a multilayer perceptron (MLP) or a multilayer transformer network, etc. The prediction network can be implemented as a fully connected layer.

[0093] Exemplarily, taking the DLANetDD3D network as an example, the algorithm model for extracting the first image feature can be preloaded with multiple weights on the depth estimation task, and different weights can support the extraction of features of images of different sizes (such as resolutions). In this way, for an image, the DLANetDD3D network, for example, supports the extraction of image sub-features of different sizes (such as resolutions). Then, in this example, the neck network of the algorithm network can also include a feature pyramid network (Feature Pyramid Networks, FPN), which can be used to perform feature fusion on multiple image sub-features of each original image to obtain the first image feature of the corresponding original image.

[0094] The following examples are combined with Figure 2A The parking area determination method of an embodiment of the present application is introduced from the perspective of the actions performed by each network.

[0095] Taking the original image as a fisheye image as an example, the vehicle obtains four fisheye images of the front, rear, left and right directions of the vehicle, and the resolution of each fisheye image is, for example, 512*1024P. After the algorithm network receives the four fisheye images, it preprocesses each fisheye image to obtain image data of each fisheye image. For any fisheye image, the preprocessing module ( Figure 2B(not shown) 512*1024P pixels can be processed according to the RGB three channels to obtain the RGB mean and RGB variance of the fisheye image, respectively. The RGB mean and RGB variance of each fisheye image are the image data of the fisheye image. The RGB mean of a fisheye image is, for example, [123.675, 116.28, 103.53], and the RGB variance is, for example, [58.395, 57.12, 57.375]. Afterwards, the RGB mean and RGB variance of the four fisheye images are input into the DLANetDD3D network.

[0096] For any fisheye image, the DLANetDD3D network processes the image data of the fisheye image and can extract multiple image sub-features of the fisheye image. The multiple image sub-features correspond to the fisheye images of different resolutions. Furthermore, the DLANetDD3D network inputs the sub-features of various sizes of each fisheye image into the FPN.

[0097] FPN can fuse sub-features of various sizes of each fisheye image, and the fused features are the image features of the corresponding fisheye image, that is, the aforementioned first image features. Afterwards, FPN transmits the image features of the four fisheye images to the feature processing network.

[0098] It should be pointed out that the fisheye image is taken by a fisheye camera, so each fisheye image is distorted. Moreover, the fisheye camera is affected by its own parameters, and the corresponding three-dimensional coordinate system is the three-dimensional coordinate system under the perspective of the fisheye camera, which does not match the BEV three-dimensional coordinate system of the vehicle. Based on this, the feature processing network is used to convert the first image feature under the perspective of the fisheye camera to the BEV area to obtain the BEV feature of the vehicle (i.e., the aforementioned second image feature).

[0099] Among them, each of the 4 fisheye images corresponds to a direction of BEV, and the feature processing network can convert the image features of each fisheye image into BEV image features of the corresponding direction through the following processing. The processing process of the feature processing network for the image features of each fisheye image is basically the same. The following takes the processing process of the image features of one fisheye image as an example to introduce the processing process of the feature processing network.

[0100] like Figure 2B As shown, the feature processing network can convert the image features of the fisheye image into BEV image features through the processing process of S201-S207.

[0101] S201, constructing a conversion relationship between the BEV coordinate system and the fisheye camera coordinate system.

[0102] For example, the coordinates of the corresponding pixel in the fisheye camera coordinate system can be calculated according to the coordinates of each pixel in the BEV coordinate system. The conversion relationship between each pixel in the BEV coordinate system and the fisheye camera coordinate system represents the conversion relationship between the BEV coordinate system and the fisheye camera coordinate system.

[0103] The conversion relationship satisfies, for example: P = RP w +T, where P refers to the pixel coordinate in the fisheye camera coordinate system, P w refers to the pixel coordinates of the vehicle BEV coordinate system, R refers to the angle rotation matrix, and T refers to the position translation matrix.

[0104] S202, calculating the coordinates of the pixels in the fisheye camera coordinate system in the rectangular three-dimensional coordinate system.

[0105] A rectangular three-dimensional coordinate system refers to a Cartesian coordinate system including three axial directions: an x-axis, a y-axis, and a z-axis. The rectangular three-dimensional coordinate system is a reference coordinate system.

[0106] This coordinate refers to the coordinate without considering the distortion (hereinafter referred to as the undistorted coordinate).

[0107] The undistorted coordinates satisfy, for example: Among them, P' new refers to the undistorted pixel, X C P' new The actual coordinate of the x-axis in the rectangular three-dimensional coordinate system, Y C P' new The actual coordinate of the y-axis in the rectangular three-dimensional coordinate system, Z C P' new The actual coordinate of the z-axis in the rectangular three-dimensional coordinate system, X P’new P' new The coordinate in the x-axis direction when it is not distorted in the rectangular three-dimensional coordinate system, Y P’new P' new The coordinate in the y-axis direction when it is not distorted in a rectangular three-dimensional coordinate system.

[0108] S203, calculating the arc length of the pixel in the rectangular three-dimensional coordinate system when the pixel is not distorted.

[0109] The arc length may refer to the radius in pixels from the origin of a rectangular three-dimensional coordinate system.

[0110] Undistorted arc length r′ new Can satisfy:

[0111] S204, calculating the incident angle θ of the pixel according to the undistorted arc length.

[0112] The incident angle θ of a pixel may satisfy, for example:

[0113] θ new = atan(r′ new )

[0114] S205, calculating the distorted arc length of the pixel according to the incident angle θ.

[0115] For example, the distortion arc length r new satisfy:

[0116]

[0117] Among them, K1, K2, K3 and K4 refer to the influencing parameters of arc length.

[0118] S206: Determine the coordinates of the pixel after orthogonal correction according to the distorted arc length.

[0119] The coordinates after orthodontics refer to the coordinates of the pixel in a rectangular three-dimensional coordinate system after orthodontics.

[0120] The coordinates after orthodontic treatment can be obtained, for example, by the following calculation:

[0121]

[0122] Among them, X Pnew It refers to the coordinate of the pixel in the x-axis direction in the rectangular three-dimensional coordinate system after orthodontic treatment. Pnew It refers to the coordinate of the pixel in the y-axis direction in the rectangular three-dimensional coordinate system after orthogonal correction.

[0123] S207, according to the conversion relationship between the BEV coordinate system and the fisheye camera coordinate system, projecting the orthogonalized pixels to the position corresponding to the BEV area corresponding to the fisheye image, to obtain the BEV image features corresponding to the fisheye image.

[0124] Each of the four fisheye images is processed according to the processing steps S201-S207 to obtain BEV image features in four directions. Afterwards, the feature processing network can fuse the BEV image features in the four directions to obtain the BEV image features of the vehicle, and transmit the BEV image features to the prediction network.

[0125] Exemplarily, for any two adjacent directions, the pixel features of the same pixels in the overlapping area of ​​the two directions are added, and the independent pixels in each direction are retained, so as to obtain the BEV image features of the vehicle.

[0126] In this implementation scenario, the prediction network can be implemented as a fully connected layer, which can be divided into an occupancy prediction module and a height prediction module. The prediction network can, for example, pass the input BEV image features to the occupancy prediction module and the height prediction module respectively. The occupancy prediction module can reduce the number of channels to 2 through two layers of 3*3 convolutions and one layer of 1*1 convolution. The two channels can, for example, respectively represent the probability value of the location being occupied. The larger the probability value, the greater the possibility of being occupied. The height prediction module can reduce the number of channels to 2 through two layers of 3*3 convolutions and one layer of 1*1 convolution. The two channels of the height prediction module respectively represent the first height value and the second height value of the location. For suspended obstacles (such as fire hydrants on the wall), the first height value is the lower edge height of the suspended obstacle, and the second height value is the upper edge height of the suspended obstacle.

[0127] It should be noted that the algorithm network called to execute the parking area determination method may be a parking area determination model that has been trained to be usable. Based on this, the embodiment of the present application may also include an implementation process of training the model to be trained to obtain the parking area determination model.

[0128] In some embodiments, the embodiments of the present application may use labeled sample images to train the training model. The sample images include multiple groups of labeled images under the first three-dimensional perspective.

[0129] The training method may include:

[0130] Call the model to be trained to identify the sample image and obtain the position prediction result and the height prediction result. Calculate the smoothing loss function (crossentropy loss) and the cross entropy loss function (smoothl1loss), and determine whether the smoothing loss function and the cross entropy loss function converge. If both the smoothing loss function and the cross entropy loss function converge, use the model to be trained as the parking area determination model; if the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the model with adjusted parameters as the new model to be trained, and call the model to be trained again to identify the sample image.

[0131] Among them, the smooth loss function (smoothl1loss) is used to characterize the loss between the type prediction result and the labeled height feature; the cross entropy loss function (crossentropy loss) is used to characterize the loss between the position prediction result and the labeled position feature.

[0132] It should be understood that Figure 2A Schematic algorithm network, and Figure 2BThe processing of each network model in the illustrated algorithm network is a schematic description and does not limit the embodiments of the present application. In other embodiments, each network model in the algorithm network can also be implemented based on other deep learning mechanisms. Accordingly, the operation process of each network model and the output feature data can be different from the presentation of the above embodiment. This is not limited here.

[0133] The present application also provides a parking area determination device, which can be deployed in an automatic parking system. The parking area determination device modularizes the above algorithm functions through software, hardware, or a combination of software and hardware, and can be used to execute the parking area determination method provided in any of the above embodiments.

[0134] like Figure 3 As shown, the device comprises:

[0135] The first feature extraction module 31 is used to extract first image features of at least two original images respectively, wherein the at least two original images include original images in various directions in the panoramic direction of the vehicle, and the first image features of any original image are features in a first three-dimensional perspective coordinate system, and the first image features of any original image include position features and height features of image content in the original image; the feature conversion module 32 is used to convert the first image features corresponding to any direction in the panoramic direction of the vehicle, and the converted features are second image features in the direction, and the second image features are features in a second three-dimensional perspective coordinate system, and the second three-dimensional perspective coordinate system refers to the coordinate system of the panoramic direction of the vehicle; the feature fusion module 33 is used to fuse the second image features in various directions in the panoramic direction of the vehicle, and the fused features are the panoramic image features of the vehicle; the second feature extraction module 34 is used to extract the position features and height features of obstacles in the area around the vehicle from the panoramic image features, and the drivable area determined by the position features and height features of the obstacles is the parking area of ​​the vehicle.

[0136] Optionally, the feature extraction module 401 is further used to extract features from the panoramic image to obtain at least two image sub-features, and the at least two image sub-features constitute the image features; the at least two image sub-features represent features of the panoramic image at at least one size.

[0137] Optionally, the feature conversion module 32 is also used to determine the conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system; for any direction of the panoramic direction of the vehicle, calculate the distortion parameters of each pixel feature in the first image feature corresponding to the direction, the distortion parameter of any pixel feature is the distortion parameter of the pixel feature relative to the reference coordinate system, and the reference coordinate system is a rectangular three-dimensional coordinate system matching the first three-dimensional viewing angle coordinate system; according to the distortion parameters of each pixel feature, determine the reference pixel feature of the corresponding pixel feature in the reference coordinate system to obtain the orthogonal image feature in the reference coordinate system; according to the conversion relationship, convert the orthogonal image feature into the image feature in the second three-dimensional viewing angle coordinate system to obtain the second image feature in the direction.

[0138] Optionally, the feature conversion module 32 is also used to calculate, for any pixel feature, the radius and incident angle of the pixel feature in the reference coordinate system; calculate the distortion incident angle based on the incident angle in the reference coordinate system; calculate the distortion arc length of the pixel feature based on the distortion incident angle, and the distortion arc length is the distortion parameter of the pixel feature.

[0139] Optionally, the position characteristics of obstacles in the area around the vehicle include: a probability value that the position is an obstacle; the height characteristics of obstacles in the area around the vehicle include: a first height value and a second height value of the obstacle, the first height value refers to the distance between the lowest point of the obstacle and the ground, and the second height value refers to the distance between the highest point of the obstacle and the ground.

[0140] Optionally, the feature fusion module 33 is further configured to add pixel features of the same pixels in an overlapping area in the second image features of any two adjacent directions in the panoramic direction of the vehicle.

[0141] Optionally, the first feature extraction module 31 is further used to perform feature extraction on any original image to obtain at least two image sub-features, where the at least two image sub-features represent features of the original image at at least one size.

[0142] Optionally, the feature fusion module 33 is further used to perform feature fusion on the at least two image sub-features, and the fused feature is the first image feature of the original image.

[0143] Optionally, the parking area determination device further includes a training module, the training module is used to train the model to be trained to obtain a parking area determination model, and the parking area determination model is used to execute the parking area determination method. The training module is also used to call the model to be trained to identify the sample image to obtain a position prediction result and a height prediction result, wherein the sample image includes multiple groups of annotated images under the first three-dimensional perspective; calculate a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the position prediction result and the annotated position feature, and the cross entropy loss function is used to characterize the loss between the type prediction result and the annotated height feature; determine whether the smoothing loss function and the cross entropy loss function converge; if the smoothing loss function and the cross entropy loss function both converge, use the model to be trained as the parking area determination model; if the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the model after the parameter adjustment as the new model to be trained, and perform the operation of calling the model to be trained to identify the sample image again.

[0144] The parking area determination device provided in the embodiment of the present application and the parking area determination method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0145] The present application also provides an electronic device for use in an automatic parking system to execute the above parking area determination method. Figure 4 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 5 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, wherein the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, the parking area determination method provided in any of the aforementioned embodiments of the present application is executed.

[0146] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0147] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store a program, and the processor 400 executes the program after receiving an execution instruction. The parking area determination method disclosed in any implementation of the embodiment of the present application may be applied to the processor 400, or implemented by the processor 400.

[0148] The processor 400 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 400. The above processor 400 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0149] The electronic device provided in the embodiment of the present application and the parking area determination method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0150] The present application also provides a computer-readable storage medium corresponding to the parking area determination method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the parking area determination method provided in any of the aforementioned embodiments will be executed.

[0151] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0152] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the parking area determination method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0153] It should be noted that:

[0154] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0155] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0156] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0157] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A parking area determination method, characterized in that: include: Extracting first image features of at least two original images respectively, wherein the at least two original images include original images in each direction in the panoramic direction of the vehicle, the first image feature of any original image is a feature in a first three-dimensional viewing angle coordinate system, and the first image feature of any original image includes a position feature and a height feature of an image content in the original image; For any direction of the panoramic direction of the vehicle, convert the first image feature corresponding to the direction, and the converted feature is the second image feature of the direction, and the second image feature is a feature in a second three-dimensional viewing angle coordinate system, and the second three-dimensional viewing angle coordinate system refers to the coordinate system of the panoramic direction of the vehicle; Performing feature fusion on the second image features in each direction of the panoramic direction of the vehicle, wherein the fused features are the panoramic image features of the vehicle; The position features and height features of obstacles in the area around the vehicle are extracted from the panoramic image features, and the drivable area determined by the position features and height features of the obstacles is the parking area of ​​the vehicle.

2. The method according to claim 1, characterized in that For any direction of the panoramic direction of the vehicle, converting the first image feature corresponding to the direction, and the converted feature is the second image feature of the direction, including: Determining a conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system; For any direction of the panoramic direction of the vehicle, calculate the distortion parameter of each pixel feature in the first image feature corresponding to the direction, wherein the distortion parameter of any pixel feature is the distortion parameter of the pixel feature relative to a reference coordinate system, and the reference coordinate system is a rectangular three-dimensional coordinate system matching the first three-dimensional viewing angle coordinate system; Determining, according to the distortion parameters of the respective pixel features, reference pixel features of the corresponding pixel features in the reference coordinate system, so as to obtain orthogonal image features in the reference coordinate system; According to the conversion relationship, the orthodontic image features are converted into image features in the second three-dimensional viewing angle coordinate system to obtain second image features in the direction.

3. The method according to claim 2, characterized in that The calculating the distortion parameter of each pixel feature in the first image feature corresponding to the direction includes: For any pixel feature, calculating the radius and the incident angle of the pixel feature in the reference coordinate system; Calculating the distorted incident angle according to the incident angle in the reference coordinate system; The distortion arc length of the pixel feature is calculated according to the distortion incident angle, and the distortion arc length is a distortion parameter of the pixel feature.

4. The method according to claim 2, characterized in that: The conversion relationship between the second three-dimensional viewing angle coordinate system and the first three-dimensional viewing angle coordinate system is used to represent at least one of the following: Any pixel coordinate in the second three-dimensional viewing angle coordinate system is converted to a rotation angle and a translation position in the first three-dimensional viewing angle coordinate system.

5. The method according to claim 1, characterized in that The position characteristics of the obstacles in the area around the vehicle include: a probability value of the position being an obstacle; The height characteristics of the obstacles in the area around the vehicle include: a first height value and a second height value of the obstacle, the first height value refers to the distance between the lowest point of the obstacle and the ground, and the second height value refers to the distance between the highest point of the obstacle and the ground.

6. The method according to claim 1, characterized in that The extracting first image features of at least two original images respectively comprises: For any original image, extract features of the original image to obtain at least two image sub-features, where the at least two image sub-features represent features of the original image at at least one size; Performing feature fusion on the at least two image sub-features, wherein the fused feature is the first image feature of the original image; The step of fusing the second image features in each direction of the panoramic direction of the vehicle includes: For any two adjacent directions of the panoramic direction of the vehicle, pixel features of the same pixels in the overlapping area of ​​the second image features in the two directions are added.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: The model to be trained is trained to obtain a parking area determination model, wherein the parking area determination model is used to execute the parking area determination method: The training method comprises: Calling the model to be trained to identify a sample image to obtain a position prediction result and a height prediction result, wherein the sample image includes a plurality of groups of annotated images under a first three-dimensional perspective; Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the type prediction result and the labeled height feature, and the cross entropy loss function is used to characterize the loss between the position prediction result and the labeled position feature; Determine whether the smoothing loss function and the cross entropy loss function converge; If both the smoothing loss function and the cross entropy loss function converge, using the to-be-trained model as the parking area determination model; If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the model with adjusted parameters as a new model to be trained, and again perform the operation of calling the model to be trained to identify the sample image.

8. A parking area determination device, characterized in that: include: A first feature extraction module is used to extract first image features of at least two original images respectively, wherein the at least two original images include original images in various directions in the panoramic direction of the vehicle, the first image feature of any original image is a feature in a first three-dimensional viewing angle coordinate system, and the first image feature of any original image includes a position feature and a height feature of the image content in the original image; a feature conversion module, for converting, for any direction of the panoramic direction of the vehicle, a first image feature corresponding to the direction, wherein the converted feature is a second image feature of the direction, wherein the second image feature is a feature in a second three-dimensional viewing angle coordinate system, wherein the second three-dimensional viewing angle coordinate system refers to a coordinate system of the panoramic direction of the vehicle; A feature fusion module, used for fusing the second image features in each direction in the panoramic direction of the vehicle, and the fused features are the panoramic image features of the vehicle; The second feature extraction module is used to extract the position features and height features of obstacles in the area around the vehicle from the panoramic image features, and the drivable area determined by the position features and height features of the obstacles is the parking area of ​​the vehicle.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images