A method and device for instance tracking
By directly calculating the image feature pyramid and mask combination coefficients, the problem of time-consuming instance tracking in existing technologies is solved, achieving more efficient instance tracking and improving the real-time processing capability of autonomous driving systems.
Patent Information
- Application Number
- CN202011454203.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-10
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-12-10
AI Technical Summary
Existing technologies for instance tracking in autonomous driving are time-consuming and inefficient, especially when detecting, segmenting, and tracking images acquired at different times.
An instance tracking method is adopted, which determines the current image from continuously acquired images by the acquisition device, inputs it into a pre-trained convolutional neural network layer to obtain an image feature pyramid, and inputs it into the first and second fully convolutional network layers respectively to determine the mask and mask combination coefficients, directly calculating the feature vector of the tracked instance and the tracking result, thus avoiding the process of selecting the region of interest.
It improves the efficiency of instance tracking, reduces the time spent on image detection, segmentation and tracking, and enhances the real-time processing capabilities of autonomous driving systems.
Smart Images

Figure CN112561961B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and more particularly to a method and device for instance tracking. Background Art
[0002] At present, with the development of unmanned driving technology, instance tracking technology has become a key research technology in the field of unmanned driving because of its help in drawing high-precision maps and determining unmanned vehicle control strategies.
[0003] Taking the application in the unmanned vehicle control scenario as an example, the unmanned vehicle can be equipped with a sensor for collecting images. When it is necessary to determine the motion strategy, the images collected at each moment in the recent period can first be instance segmented, and the same instances in each image after instance segmentation can be associated to achieve detection, segmentation and tracking of instances in each image. Based on the association relationship between each image after instance segmentation and the same instances in each image, the motion strategy of the unmanned vehicle can be determined later.
[0004] In the prior art, a commonly used method for detecting, segmenting and tracking instances in each image is implemented based on a model of a network structure of MaskTrack-Region Convolutional Neural Networks (MaskTrack-RCNN).
[0005] Specifically, first, multiple frames of images containing the same instance are input, and each image is subjected to feature extraction by Convolutional Neural Networks (CNN), and the region of interest (ROI) is selected. Then, each ROI region is calibrated through the ROI Align operation, so that the feature sizes extracted from instances of different sizes are consistent. The features will be input into the BBox layer, Mask layer, and tracking layer respectively. Among them, the BBox layer is used to determine the size and category prediction of the bounding box of the tracking instance, the Mask layer is used to predict the segmentation mask of the tracking instance, and the tracking layer realizes the tracking and segmentation of the tracking instances in different images by calculating the relationship between the features of each tracking instance between different images. The motion strategy of the unmanned vehicle can be determined based on the tracking results later.
[0006] However, when the prior art realizes the purpose of detecting, segmenting and tracking instances in each image, it is necessary to go through the stage of inputting the image into the convolutional neural network for feature extraction and selecting the region of interest, and the stage of inputting the proofread features into the BBox layer, the Mask layer and the tracking layer by ROI Align. As a result, in actual applications, the prior art consumes a lot of time in realizing the detection, segmentation and tracking of instances in images collected at different times, resulting in low efficiency. Summary of the invention
[0007] This specification provides a method and device for instance tracking to partially solve the above-mentioned problems existing in the prior art.
[0008] This manual adopts the following technical solutions:
[0009] This specification provides an instance tracking method, including:
[0010] Determine, from each frame of images continuously acquired by the acquisition device, an image for inputting into the classification model at the current moment;
[0011] Inputting the image into the convolutional neural network layer of the pre-trained classification model to obtain the image feature pyramid at the current moment;
[0012] Taking the image feature pyramid at the current moment as input, respectively inputting it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model, obtaining masks corresponding to each feature dimension output by the first fully convolutional network layer, and mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer;
[0013] Determine the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer;
[0014] Determining a feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment;
[0015] The tracking result of the tracking instance is determined according to the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at a specified historical moment.
[0016] Optionally, the image is input into a convolutional neural network layer of a pre-trained classification model to obtain an image feature pyramid at the current moment, specifically including:
[0017] Inputting the image into a convolutional neural network layer of a pre-trained classification model to obtain an image feature vector at the current moment;
[0018] The image feature pyramid at the current moment is determined according to the image feature vector at the current moment.
[0019] Optionally, the image feature pyramid consists of several layers of feature maps;
[0020] The image feature pyramid at the current moment is used as input, and is respectively input into the first fully convolutional network layer and the second fully convolutional network layer of the classification model to obtain masks corresponding to each feature dimension output by the first fully convolutional network layer and mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer, specifically including:
[0021] Taking at least one layer of feature graph of the image feature pyramid at the current moment as input, inputting it into the first fully convolutional network layer of the classification model, and obtaining masks corresponding to each feature dimension output by the first fully convolutional network layer;
[0022] The image feature pyramid at the current moment is used as input and input into the second fully convolutional network layer of the classification model to obtain mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer.
[0023] Optionally, the number of the mask combination coefficients is equal to the number of masks corresponding to each feature dimension;
[0024] Determining the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer, specifically includes:
[0025] For each tracking instance, the masks corresponding to each feature dimension and the mask combination coefficient corresponding to the tracking instance are linearly combined to determine the mask corresponding to the tracking instance.
[0026] Optionally, according to the feature vectors of each tracking instance at the current moment and the tracking targets determined at the specified historical moment, for each tracking instance, the feature vector of the tracking instance at the current moment is matched with the feature vectors of each tracking instance to be matched determined at the specified historical moment, and the similarity between the feature vector of the tracking instance at the current moment and the feature vectors of each tracking instance to be matched determined at the specified historical moment is determined;
[0027] Determine, from the tracking instances to be matched at the specified historical moment, a tracking instance to be matched that is associated with the tracking instance according to the determined similarity and a preset threshold;
[0028] According to each tracking instance at the current moment and the determined tracking instance to be matched and associated with each tracking instance, the tracking result of each tracking instance is determined.
[0029] Optionally, the classification model includes: a convolutional neural network layer, a first fully convolutional network layer, a second fully convolutional network layer, an assembly layer and a fully connected layer, and the classification model is trained by the following method:
[0030] Obtain several images collected by the collection device in the past as training samples, and determine the classification label of each instance in each image as the sample label;
[0031] Inputting the determined training samples into a convolutional neural network layer of a classification model to be trained to obtain an image feature pyramid of the training samples;
[0032] Taking the image feature pyramid of the training sample as input, respectively inputting it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model to be trained, obtaining the mask corresponding to each feature dimension output by the first fully convolutional network layer, and the mask combination coefficient corresponding to each tracking instance in the training sample output by the second fully convolutional network layer;
[0033] Inputting the mask combination coefficients corresponding to each feature dimension output by the first fully convolutional network layer and each tracking instance in the training sample output by the second fully convolutional network layer into the assembly layer of the classification model to be trained, so as to obtain the mask corresponding to each tracking instance in the training sample;
[0034] For each mask corresponding to the tracking instance in the training sample, input the mask corresponding to the tracking instance into the fully connected layer of the classification model to be trained to obtain the classification result of the tracking instance in the training sample;
[0035] The classification model is trained with the optimization goal of minimizing the deviation between the classification result and the sample label.
[0036] Optionally, determining a feature vector of each tracking instance at the current moment according to a mask corresponding to each tracking instance at the current moment specifically includes:
[0037] The second fully convolutional network layer is also used to output the bounding box of each tracking instance respectively for each tracking instance;
[0038] For each tracking instance, a feature vector of the tracking instance at the current moment is determined according to the bounding box of the tracking instance and the mask corresponding to the tracking instance at the current moment.
[0039] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned instance segmentation method is implemented.
[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned instance segmentation method when executing the program.
[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:
[0042] In the instance segmentation method provided in the present specification, the image input into the classification model at the current moment can be determined from a number of images collected by the collection device, and then the image is input into the convolutional neural network layer of the pre-trained classification model to obtain the image feature pyramid at the current moment, and the image feature pyramid at the current moment is respectively input into the first full convolutional network layer and the second full convolutional network layer of the classification model to obtain the mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance, and the mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance are combined to obtain the mask corresponding to each tracking instance at the current moment, and the feature vector of each tracking instance at the current moment is determined according to the mask corresponding to each tracking instance at the current moment, and the tracking result of the tracking instance is determined according to the determined feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at the specified historical moment.
[0043] It can be seen from the above method that, unlike the prior art method of first selecting the region of interest and then processing the region of interest, this method does not need to determine the region of interest, so it takes less time to detect, segment and track the image, further improving the efficiency of instance tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:
[0045] Figure 1 A flowchart of the method for example tracking provided in this specification;
[0046] Figure 2 A schematic diagram of the convolutional neural network layers of the classification model provided in this specification;
[0047] Figure 3 The first fully convolutional network layer and the second fully convolutional network layer of the classification model provided in this specification;
[0048] Figure 4 A schematic diagram of a determination mask provided for this specification;
[0049] Figure 5A schematic diagram of determining a feature vector corresponding to a tracking instance provided in this specification;
[0050] Figure 6 A schematic diagram of determining a tracking instance to be matched associated with a tracking instance provided in this specification;
[0051] Figure 7 A schematic diagram of an apparatus for example tracking provided in this specification;
[0052] Figure 8 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0054] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0055] Figure 1 The example segmentation method provided in this specification is a flowchart, which specifically includes the following steps:
[0056] S100: Determine, from the frames of images continuously acquired by the acquisition device, an image used for inputting into a classification model at the current moment.
[0057] At present, instance tracking technology can be applied in many fields, especially in the field of unmanned driving technology. The instance tracking method can be used to assist in determining the motion trajectory of obstacles around the unmanned vehicle, so as to determine the motion strategy of the unmanned vehicle based on the determined motion trajectory. It can also be used in the creation of high-precision map scenes. Determine the correspondence between instances in the image, and combine with the semantic model to determine the objects in the image that can be used to build the map (for example, buildings, traffic signs and other fixed objects). In addition, since instance tracking technology is usually used in the field of unmanned driving technology, the instance tracking method can generally be executed by an unmanned vehicle or a background server communicating with the unmanned vehicle. For the convenience of description, this specification takes the server executing the instance tracking method and the application of the instance tracking method to determine the scene of the unmanned vehicle's motion strategy as an example for subsequent explanation.
[0058] In one or more embodiments provided in this specification, when the server needs to perform instance tracking, it can first obtain images collected by the acquisition device. The acquisition device can be an image sensor set on the unmanned vehicle, which is used to collect images around the unmanned vehicle. In addition, the frame rate of the image collected by the acquisition device can be set as needed, such as 24 frames per second (Frame Per Second, FPS), 60FPS, and this specification does not limit this.
[0059] Specifically, the server can determine the image used to input the classification model at the current moment from each frame of images continuously collected by the acquisition device. Of course, this manual does not limit how the server specifically obtains the image, and it can be set as needed. For example, the unmanned vehicle can transmit the image to the server in real time after the acquisition device collects the image. When the server needs to track the instance, it can determine the image used to input the classification model at the current moment from the stored images. Alternatively, the server can also send an acquisition request to the unmanned vehicle, so that the unmanned vehicle determines the image used to input the classification model at the current moment from the image collected by the acquisition device, and returns the determined image to the server.
[0060] In addition, in the scenario of determining the control strategy of an unmanned vehicle, the strategy of the unmanned vehicle at the next moment is usually determined based on the tracking results of the instances in the image collected by the unmanned vehicle at the current moment. Therefore, the image collected by the collection device at the current moment can be used as the image input into the classification model at the current moment.
[0061] It should be noted that the unmanned vehicle mentioned in this specification may refer to unmanned vehicles, robots, automatic delivery equipment and other equipment that can achieve automatic driving. Based on this, the unmanned vehicle using the instance segmentation method provided in this specification can be used to perform delivery tasks in the delivery field, such as using unmanned driving equipment for delivery business scenarios such as express delivery, logistics, and takeaway delivery.
[0062] S102: Input the image into a convolutional neural network layer of a pre-trained classification model to obtain an image feature pyramid at a current moment.
[0063] In one or more embodiments provided in this specification, after receiving an image used to input a classification model at the current moment, the server may input the image into a convolutional neural network layer of a pre-trained classification model to obtain an image feature pyramid at the current moment.
[0064] Specifically, in this specification, the convolutional neural network layer of the classification model can be as follows Figure 2 As shown, Figure 2The schematic diagram of the convolutional neural network layer of the classification model provided in this specification, the server can first input the image received at the current moment for inputting the classification model into the convolutional neural network layer (Convolutional Neural Networks, CNN) of the classification model, and through the multiple convolutional layers in the CNN, feature maps with different image dimensions (W×H) and depths (C) can be obtained respectively, and the image feature vector at the current moment is determined according to the feature map output by the specified convolutional layer, and the feature maps corresponding to the image feature pyramid at the current moment are determined according to at least part of the feature maps contained in the image feature vector at the current moment. It should be noted that the feature map constituting the image feature vector is different from the feature map constituting the image feature pyramid.
[0065] Furthermore, when determining each feature map of the image feature pyramid at the current moment by the image feature vector at the current moment, methods such as convolution and bilinear interpolation may be used.
[0066] For example, in Figure 2 In , the image input to the classification model at the current moment passes through the five convolutional layers of CNN to obtain five feature maps with different image dimensions (W×H) and depths (C), which are recorded as C1, C2, C3, C4, and C5 respectively. After determining these five feature maps, the server can also determine the image feature vector at the current moment based on the five feature maps, and the image feature vector contains at least part of the feature maps in the five determined feature maps. Figure 2 In the figure, the image feature vector at the current moment includes three feature maps, C3, C4, and C5. C5 can be input into the convolution layer to obtain P5. P4 can be obtained by adding the result of bilinear interpolation of P5 and the result of inputting C4 into the convolution layer. P3 can be obtained by adding the result of bilinear interpolation of P4 and the result of inputting C3 into the convolution layer. P5 can be input into the convolution layer to obtain P6. P6 can be input into the convolution layer to obtain P7. Therefore, the five feature maps P3, P4, P5, P6, and P7 determined can form the image feature pyramid at the current moment, and the number of layers of the image feature pyramid is 5.
[0067] It should be noted that the method used to determine the image feature pyramid by the image feature vector, the convolution kernel size, etc. mentioned in this specification can be set as needed, and the convolution layers of C5, C4, P5, and P6 inputs can be the same or different, and can be set according to specific needs. This specification does not impose any restrictions on this.
[0068] S104: Taking the image feature pyramid at the current moment as input, respectively inputting it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model, obtaining masks corresponding to each feature dimension output by the first fully convolutional network layer, and mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer.
[0069] In one or more embodiments provided in the present specification, after determining the image feature pyramid at the current moment, the image feature pyramid at the current moment can also be used as input and input into the first fully convolutional network layer and the second fully convolutional network layer of the classification model respectively to obtain the masks corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer.
[0070] Specifically, in this specification, the first fully convolutional network layer and the second fully convolutional network layer of the classification model can be as follows: Figure 3 shown.
[0071] Figure 3 For the first fully convolutional network layer and the second fully convolutional network layer of the classification model provided in this specification, the server can use the determined image feature pyramid (P3~P7) at the current moment as input, and input it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model respectively. After receiving the feature map of the image feature pyramid, the first fully convolutional network layer can obtain the mask X1~X1 of k feature dimensions according to the feature map contained in the image feature pyramid at the current moment by upsampling. k . The second fully convolutional network layer can determine the classification of each tracking instance (for example, people, vehicles, etc.) and generate a mask combination coefficient corresponding to the tracking instance, such as the prediction head branch in the You Only Look At Coefficients (YOLACT) algorithm. The mask combination coefficient can be used to combine with the mask of k feature dimensions generated by the first fully convolutional network layer to generate a mask corresponding to the tracking instance. The tracking instance is an instance that exists in the image captured by the acquisition device that inputs the classification model. The tracking instance can be a specified instance in the image captured by the acquisition device (such as people, vehicles, etc.), or it can be each instance that appears in the image. The specific scope of the tracking instance can be set as needed, and this manual does not limit this. Figure 3 The different color areas in the mask corresponding to the different feature dimensions shown in represent different feature values.
[0072] Furthermore, in order to improve processing efficiency, the server may input at least one layer of the image feature pyramid at the current moment into the first fully convolutional network layer for processing. Generally, the larger the image, the greater the resolution, the easier it is to obtain masks of different feature dimensions, and the higher the clarity of the obtained mask. Therefore, only the feature map with the largest specification, that is, P3 of the image feature pyramid at the current moment, may be selected and input into the first fully convolutional network layer of the classification model to obtain masks of different feature dimensions.
[0073] S106: Determine the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer.
[0074] In one or more embodiments provided in this specification, after determining the mask combination coefficient corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance, the server can combine the mask corresponding to each feature dimension output by the first fully convolutional network layer of the classification model with the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer to determine the mask corresponding to each tracking instance at the current moment.
[0075] In this specification, the masks corresponding to the feature dimensions of different tracking instances in the same image are the same, and the masks corresponding to the different tracking instances can be determined by different mask combination coefficients. Figure 4 shown.
[0076] Figure 4 The schematic diagram of determining the mask provided in this specification, after determining the mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance, the server can input the determined mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance into the predetermined mask generation function Among them, k refers to the mask with k dimensions output by the first fully convolutional network layer of the classification model, then X1~X k is the mask corresponding to the k feature dimensions output by the first fully convolutional network layer, a1~a k is the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer. The value of β can be set as needed to obtain the mask corresponding to each tracking instance. Figure 3 similar, Figure 4 Different colors in the mask represent different feature values.
[0077] Furthermore, linear combinations often take less time to process than nonlinear combinations. Therefore, in order to improve processing efficiency, for each tracking instance, the masks corresponding to the obtained feature dimensions and the mask combination coefficients corresponding to the tracking instance can be linearly combined, that is, β in the mask generation function is set to 1, and the mask corresponding to the tracking instance is determined.
[0078] S108: Determine a feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment.
[0079] In one or more embodiments provided in this specification, after the mask corresponding to each tracking instance at the current moment is determined, the feature vector of each tracking instance at the current moment may be determined according to the determined mask corresponding to each tracking instance at the current moment.
[0080] Specifically, in this specification, the server may convert the mask matrix corresponding to the tracking instance into a row vector or a column vector according to the mask corresponding to each tracking instance at the current moment determined in step S106, and use the vector as the feature vector of the tracking instance.
[0081] Furthermore, the mask determined in step S106 may not only include the tracking instance, but may also include the noise around the tracking instance, etc. In order to improve the purity of the determined tracking instance, the second fully convolutional network layer of the classification model can also be used to generate a bounding box for each tracking instance. Then in step S106, the server inputs the image feature pyramid at the current moment determined in step S104 into the second fully convolutional network layer of the classification model, and can also determine the bounding box of each tracking instance. After determining the bounding box for each tracking instance and the mask corresponding to each tracking instance, the server can determine the feature vector of the tracking instance at the current moment for each tracking instance based on the bounding box of the tracking instance and the mask corresponding to the tracking instance at the current moment. Figure 5 As shown, Figure 5 This is a schematic diagram of determining the feature vector corresponding to a tracking instance provided in this specification. Different colors in the image represent different feature values. The image on the left is the mask corresponding to the determined tracking instance, which includes the tracking instance (car) and background noise. For the convenience of description, it is assumed that the feature value corresponding to the pixel points in the tracking instance area in the figure is 1.2, and the feature value corresponding to the pixel points in the background noise area is 0.7. When determining the feature vector corresponding to the tracking instance based on the mask represented by the image on the left, there will be more feature values of background noise, which will be used as the feature vector corresponding to the tracking instance, resulting in the feature vector corresponding to the tracking instance containing more feature values of background noise. In order to reduce the impact of background noise on the feature vector corresponding to the tracking instance, in this specification, the server can be based on the bounding box corresponding to the tracking instance (that is, Figure 5The noise outside the bounding box is filtered out to obtain Figure 5 The right image in . Among them, the middle image contains the mask corresponding to the tracking instance and the bounding box corresponding to the tracking instance, and the dotted line is the bounding box for the tracking instance (car). The pixels outside the bounding box can be regarded as pixels that are not in the tracking instance area, so the eigenvalues of the pixels outside the bounding box can be filtered out, and a purer right image than the left image can be obtained. In the right image, the eigenvalues corresponding to the pixels of the background part outside the bounding box are 0, and only the eigenvalues corresponding to the pixels of the tracking instance area within the bounding box (i.e., 1.2) and the eigenvalues corresponding to the pixels of a small part of the background noise area (i.e., 0.7) remain. It can be seen that determining the eigenvector corresponding to the tracking instance based on the right image will greatly reduce the eigenvalues of the background noise contained in the eigenvector, making the eigenvector more accurate.
[0082] In addition, in this specification, a third fully convolutional network may be preset in the classification model, and the third fully convolutional network is used to output a bounding box for each tracking instance. The server may input the image feature pyramid at the current moment determined in step S104 into the third fully convolutional network layer of the classification model to determine the bounding box of each tracking instance, and when it is necessary to determine the feature vector of each tracking instance in the future, the bounding box and mask corresponding to the tracking instance are used as input.
[0083] S110: Determine the tracking result of the tracking instance according to the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at a specified historical moment.
[0084] In one or more embodiments provided in this specification, after determining the feature vector of each tracking instance at the current moment, the server can determine the tracking result of each tracking instance based on the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at the specified historical moment.
[0085] Specifically, in the present specification, the server may match each tracking instance according to the feature vector of the tracking instance at the current moment with the feature vectors of each tracking instance to be matched determined at a specified historical moment. The specified historical moment may include only one historical moment, such as the previous moment, or may include multiple historical moments. The specific content included in the specified historical moment is set as needed, and this specification does not impose any restrictions on this.
[0086] By matching the feature vector of the tracking instance at the current moment with the feature vectors of each tracking instance to be matched determined at the specified historical moment, the similarity between the feature vector of the tracking instance at the current moment and the feature vectors of each tracking instance to be matched determined at the specified historical moment can be obtained respectively. The tracking instance to be matched is the tracking instance contained in the image input into the classification model at the specified historical moment.
[0087] It should be noted that the classification model in this specification outputs the classification result for each tracking instance, and the classification result can be a name, for example, car 1, car 2, etc. And the name is only used to distinguish different tracking instances in the current image, and does not represent the correlation between tracking instances in images at two different moments, that is, the name of the tracking instance at the current moment may be car 1, and the name of the tracking instance to be matched associated with the tracking instance at a specified historical moment is car 3, and the tracking instance to be matched with the name car 1 at the historical moment is associated with another tracking instance at the current moment. In this specification, whether each tracking instance at the current moment and each tracking instance to be matched at the specified historical moment are associated is determined based on the similarity between the feature vectors.
[0088] After determining the similarity between the feature vector of the tracking instance at the current moment and the feature vector of each tracking instance to be matched determined at the specified historical moment, the tracking instance to be matched associated with the tracking instance can be determined from each tracking instance to be matched at the specified historical moment according to the determined similarity and the preset threshold. If the determined similarity is not less than the preset threshold, it can be determined that the tracking instance to be matched is associated with the tracking instance. If the determined similarity is less than the preset threshold, it can be determined that the tracking instance to be matched is not associated with the tracking instance. The specific value of the preset threshold can be set as needed, and this specification does not limit this.
[0089] After determining the tracking instances to be matched associated with each tracking instance, the server may determine the tracking result of each tracking instance according to each tracking instance at a current moment and the tracking instances to be matched associated with each tracking instance.
[0090] Furthermore, when more than one tracking instance to be matched is determined, that is, there are multiple tracking instances to be matched and the similarity with the tracking instance is not less than a preset threshold, the tracking instance to be matched associated with the tracking instance can also be determined according to the position of each tracking instance to be matched, such as Figure 6 shown.
[0091] Figure 6A schematic diagram for determining the tracking instance to be matched associated with the tracking instance provided in this specification, based on the position of the tracking instance vehicle 1 to be matched at the specified historical moment, the position of the tracking instance vehicle 2 to be matched at the specified historical moment and the position of the tracking instance vehicle 1 at the current moment, it can be determined that the tracking instance vehicle 1 at the current moment is associated with the tracking instance vehicle 2 to be matched at the specified historical moment, and the tracking instance moves from the position of the tracking instance vehicle 2 to be matched at the specified historical moment to the position of the tracking instance vehicle 1 at the current moment.
[0092] based on Figure 1 The instance segmentation method can determine the image input to the classification model at the current moment from a number of images collected by the collection device, and then input the image to the convolutional neural network layer of the pre-trained classification model to obtain the image feature pyramid at the current moment, and input the image feature pyramid at the current moment into the first full convolutional network layer and the second full convolutional network layer of the classification model respectively to obtain the mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance, and combine the mask corresponding to each feature dimension and the mask combination coefficient corresponding to each tracking instance to obtain the mask corresponding to each tracking instance at the current moment, and determine the feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment, and determine the tracking result of the tracking instance according to the determined feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at the specified historical moment. Different from the method of selecting the region of interest first and then processing the region of interest in the prior art, the present method does not need to determine the region of interest, so it takes less time to detect, segment and track the image, and further improves the efficiency of instance tracking.
[0093] Furthermore, during the driving process of the unmanned vehicle, there is a situation where a certain instance appears for the first time in the image collected by the image sensor, for example, when waiting for the red light at the intersection, a vehicle enters the image from outside the field of view. Therefore, in step S110, when there is a new tracking instance in the image collected by the acquisition device, that is, a tracking instance that is not included in the image collected at the historical moment, there will be a situation where the similarity between the tracking instance and each tracking instance to be matched is less than a preset threshold, then it can be determined that the tracking instance has no associated tracking instance to be matched.
[0094] In addition, the classification model used in this specification includes: a convolutional neural network layer, a first fully convolutional network layer, a second fully convolutional network layer, an assembly layer and a fully connected layer. When training the model, several images collected by the acquisition device in the past can be obtained as training samples, and the classification label of each instance in each image can be determined as a sample label. The determined training sample is input into the convolutional neural network layer of the classification model to be trained to obtain the image feature pyramid of the training sample. The image feature pyramid of the training sample is used as input, and is respectively input into the first fully convolutional network layer and the second fully convolutional network layer of the classification model to be trained to obtain the mask corresponding to each feature dimension output by the first fully convolutional network layer, and the mask combination coefficient corresponding to each tracking instance in the training sample output by the second fully convolutional network layer. The mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance in the training sample output by the second fully convolutional network layer are input into the assembly layer of the classification model to be trained to obtain the mask corresponding to each tracking instance in the training sample. For each mask corresponding to the tracking instance in the training sample, the mask corresponding to the tracking instance is input into the fully connected layer of the classification model to be trained to obtain the classification result of the tracking instance in the training sample. The classification model is trained with the optimization goal of minimizing the deviation between the classification result and the sample label.
[0095] In addition, in the prior art, the method of implementing instance tracking based on the model of the network structure of MaskTrack-RegionConvolutional Neural Networks (MaskTrack-RCNN) often requires that multiple images containing the same instance be used as training samples at the same time when training the model, and the tracking results between each tracking instance in the multiple images be used as sample labels. The determined training samples are input into the model to be trained to obtain the tracking results of the tracking instances in the training samples, and the model is trained with minimizing the deviation between the tracking results and the sample labels as the optimization goal. However, when training the classification model, the present method only needs to input a single image and determine the classification results of each tracking instance in the single image, so that the cost of creating training samples is low, the complexity of the training process is reduced, the resource consumption of the training process is low, and the training efficiency is improved.
[0096] In one or more embodiments provided in this specification, the above example tracking method is applied to the scenario of determining the motion strategy of an unmanned vehicle. Generally, the example tracking method can also be applied to the map construction scenario, and the tracking results of each instance in the image can be determined according to the image collected by the acquisition device, and a more accurate map can be constructed based on the tracking results of each instance. The acquisition device can also be an image sensor mounted on a vehicle, mounted on an unmanned vehicle, or handheld.
[0097] The above is an instance segmentation method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding instance segmentation device, such as Figure 7 shown.
[0098] The acquisition module 200 is used to determine the image used for inputting the classification model at the current moment from each frame of images continuously acquired by the acquisition device.
[0099] The first determination module 202 is used to input the image into the convolutional neural network layer of the pre-trained classification model to obtain the image feature pyramid at the current moment.
[0100] The full convolution module 204 is used to take the image feature pyramid as input and input it into the first full convolution network layer and the second full convolution network layer of the classification model respectively, to obtain the mask corresponding to each feature dimension output by the first full convolution network layer, and the mask combination coefficient corresponding to each tracking instance output by the second full convolution network layer.
[0101] The combination module 206 is used to determine the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer.
[0102] The second determination module 208 determines the feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment, and determines the tracking result of the tracking instance according to the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at the specified historical moment.
[0103] The device also includes:
[0104] The training module 210 is used to obtain a number of images collected by the collection device in the past as training samples, and determine the classification label of each instance in each image as the sample label, input the determined training samples into the convolutional neural network layer of the classification model to be trained to obtain the image feature pyramid of the training samples, and input the image feature pyramid of the training samples as input to the first full convolutional network layer and the second full convolutional network layer of the classification model to be trained, respectively, to obtain the mask corresponding to each feature dimension output by the first full convolutional network layer, and the mask combination coefficient corresponding to each tracking instance in the training sample output by the second full convolutional network layer, and the mask combination coefficient corresponding to each tracking instance in the training sample output by the first full convolutional network layer. The mask combination coefficient corresponding to each feature dimension and the mask corresponding to each tracking instance in the training sample output by the second fully convolutional network layer are input into the assembly layer of the classification model to be trained to obtain the mask corresponding to each tracking instance in the training sample. For each tracking instance in the training sample, the mask corresponding to the tracking instance is input into the fully connected layer of the classification model to be trained to obtain the classification result of the tracking instance in the training sample. The classification model is trained with the optimization goal of minimizing the deviation between the classification result and the sample label. The classification model includes: a convolutional neural network layer, a first fully convolutional network layer, a second fully convolutional network layer, an assembly layer and a fully connected layer.
[0105] Optionally, the first determination module 202 is specifically configured to input the image into a convolutional neural network of a pre-trained classification model to obtain an image feature vector at a current moment, and determine the image feature pyramid at the current moment based on the image feature vector at the current moment.
[0106] Optionally, the image feature pyramid is composed of several layers of feature maps, and the full convolution module 204 is used to take at least one layer of feature map of the image feature pyramid at the current moment as input, input it into the first full convolutional network layer of the classification model, and obtain the mask corresponding to each feature dimension output by the first full convolutional network layer, take the image feature pyramid at the current moment as input, input it into the second full convolutional network layer of the classification model, and obtain the mask combination coefficient corresponding to each tracking instance output by the second full convolutional network layer.
[0107] Optionally, the number of the mask combination coefficients is equal to the number of masks corresponding to each feature dimension. The combination module 206 is specifically used to linearly combine the masks corresponding to each feature dimension and the mask combination coefficients corresponding to the tracking instance for each tracking instance to determine the mask corresponding to the tracking instance.
[0108] Optionally, the second determination module 208 is used to match, for each tracking instance, the feature vector of the tracking instance at the current moment with the feature vector of each tracking instance to be matched determined at the specified historical moment, determine the similarity between the feature vector of the tracking instance at the current moment and the feature vector of each tracking instance to be matched determined at the specified historical moment, determine the tracking instance to be matched associated with the tracking instance from the tracking instances to be matched at the specified historical moment based on the determined similarity and a preset threshold, and determine the tracking result of each tracking instance based on the tracking instances at the current moment and the determined tracking instances to be matched associated with each tracking instance.
[0109] Optionally, the second determination module 208, specifically used in the second fully convolutional network layer, is also used to output the bounding box of each tracking instance respectively, and for each tracking instance, determine the feature vector of the tracking instance at the current moment based on the bounding box of the tracking instance and the mask corresponding to the tracking instance at the current moment.
[0110] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides instance tracking methods.
[0111] This manual also provides Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8 As mentioned above, at the hardware level, the unmanned driving device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0112] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0113] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0114] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0115] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0116] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0117] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0120] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0121] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0122] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0123] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0124] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0125] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0126] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0127] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.
Claims
1. A method for instance tracking, characterized in that: The method comprises: Determine, from each frame of images continuously acquired by the acquisition device, an image for inputting into the classification model at the current moment; Inputting the image into a convolutional neural network layer of a pre-trained classification model to obtain an image feature pyramid at the current moment; Taking the image feature pyramid at the current moment as input, respectively inputting it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model, obtaining masks corresponding to each feature dimension output by the first fully convolutional network layer, and mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer; Determine the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer; Determining a feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment; Determining a tracking result of a tracking instance according to the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance to be matched determined at a specified historical moment; The classification model includes: a convolutional neural network layer, the first fully convolutional network layer, the second fully convolutional network layer, an assembly layer and a fully connected layer, and the classification model is trained by the following method: Obtain several images collected by the collection device in the past as training samples, and determine the classification label of each instance in each image as the sample label; Inputting the determined training samples into a convolutional neural network layer of a classification model to be trained to obtain an image feature pyramid of the training samples; Taking the image feature pyramid of the training sample as input, respectively inputting it into the first fully convolutional network layer and the second fully convolutional network layer of the classification model to be trained, obtaining the mask corresponding to each feature dimension output by the first fully convolutional network layer, and the mask combination coefficient corresponding to each tracking instance in the training sample output by the second fully convolutional network layer; Inputting the mask combination coefficients corresponding to each feature dimension output by the first fully convolutional network layer and each tracking instance in the training sample output by the second fully convolutional network layer into the assembly layer of the classification model to be trained, so as to obtain the mask corresponding to each tracking instance in the training sample; For each mask corresponding to the tracking instance in the training sample, input the mask corresponding to the tracking instance into the fully connected layer of the classification model to be trained to obtain the classification result of the tracking instance in the training sample; The classification model is trained with the optimization goal of minimizing the deviation between the classification result and the sample label.
2. The method according to claim 1, characterized in that The image is input into the convolutional neural network layer of the pre-trained classification model to obtain the image feature pyramid at the current moment, specifically including: Inputting the image into a convolutional neural network layer of a pre-trained classification model to obtain an image feature vector at the current moment; The image feature pyramid at the current moment is determined according to the image feature vector at the current moment.
3. The method according to claim 1, characterized in that The image feature pyramid is composed of several layers of feature maps; The image feature pyramid at the current moment is used as input, and is respectively input into the first fully convolutional network layer and the second fully convolutional network layer of the classification model to obtain masks corresponding to each feature dimension output by the first fully convolutional network layer and mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer, specifically including: Taking at least one layer of feature graph of the image feature pyramid at the current moment as input, inputting it into the first fully convolutional network layer of the classification model, and obtaining masks corresponding to each feature dimension output by the first fully convolutional network layer; The image feature pyramid at the current moment is used as input and input into the second fully convolutional network layer of the classification model to obtain mask combination coefficients corresponding to each tracking instance output by the second fully convolutional network layer.
4. The method according to claim 1, characterized in that The number of the mask combination coefficients is equal to the number of masks corresponding to each feature dimension; Determining the mask corresponding to each tracking instance at the current moment according to the mask corresponding to each feature dimension output by the first fully convolutional network layer and the mask combination coefficient corresponding to each tracking instance output by the second fully convolutional network layer, specifically includes: For each tracking instance, the masks corresponding to each feature dimension and the mask combination coefficient corresponding to the tracking instance are linearly combined to determine the mask corresponding to the tracking instance.
5. The method according to claim 1, characterized in that Determining the tracking result of the tracking instance according to the feature vector of each tracking instance at the current moment and the feature vector of the tracking instance determined at the specified historical moment specifically includes: For each tracking instance, matching the feature vector of the tracking instance at the current moment with the feature vectors of each tracking instance to be matched determined at the specified historical moment, and determining the similarity between the feature vector of the tracking instance at the current moment and the feature vectors of each tracking instance to be matched determined at the specified historical moment; Determine, from the tracking instances to be matched at the specified historical moment, a tracking instance to be matched that is associated with the tracking instance according to the determined similarity and a preset threshold; According to each tracking instance at the current moment and the determined tracking instance to be matched and associated with each tracking instance, the tracking result of each tracking instance is determined.
6. The method according to claim 1, characterized in that Determining the feature vector of each tracking instance at the current moment according to the mask corresponding to each tracking instance at the current moment specifically includes: The second fully convolutional network layer is also used to output the bounding box of each tracking instance respectively for each tracking instance; For each tracking instance, a feature vector of the tracking instance at the current moment is determined according to the bounding box of the tracking instance and the mask corresponding to the tracking instance at the current moment.
7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.
