A method, device and electronic equipment for detecting a target based on a convolutional neural network

Abstract

The application provides a method and device for detecting a target based on a convolutional neural network and an electronic device, to solve the problem of low detection accuracy of different size targets in the current convolutional neural network by downsampling and setting the same weight for each target. The method comprises: inputting a to-be-detected image into the convolutional neural network, and determining a target in the to-be-detected image and first information of the target based on the Nth feature map in the convolutional neural network; wherein the number of feature maps in the convolutional neural network is N; the first information comprises a first position and a first size; in response to the first size being less than or equal to a first preset value, marking the target as a first target, and determining the first target and second information of the first target in the (N-p)th feature map based on the first position; and outputting the second information of the first target.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for detecting targets based on a convolutional neural network. Background Technology

[0002] Currently, object detection is mostly done using convolutional neural network models to detect targets in scene images. Generally, when processing images, convolutional detection networks perform downsampling sequentially and obtain corresponding feature maps in order, identifying and outputting the target in the deepest feature map.

[0003] However, in the process of processing image features, as the convolutional neural network model deepens and the number of convolutional kernels increases, the receptive field also increases. When the target size in the image is small, or when it contains both large and small targets, the small target, due to its small pixel size, exhibits a significantly larger error between its receptive field and its true coordinates after multiple downsampling.

[0004] Furthermore, during the training process, when the aforementioned convolutional neural network identifies targets using methods such as IoU (Intersection over Union), smaller targets are more likely to be misidentified than larger targets due to their smaller pixel count, even with the same detection error. Therefore, when there are multiple targets with significantly different sizes to be identified, convolutional detection neural networks suffer from low detection accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for detecting targets based on a convolutional neural network, which solves the problem of low detection accuracy for targets of different sizes when downsampling is used in current convolutional neural networks and the weights of each target are set to be the same.

[0006] In a first aspect, embodiments of this application provide a method for detecting targets based on a convolutional neural network, including:

[0007] The image to be detected is input into the convolutional neural network, and the target in the image to be detected and the first information of the target are determined based on the Nth feature map in the convolutional neural network; wherein, the number of feature maps in the convolutional neural network is N, and N is a natural number greater than 2; the first information includes a first position and a first size;

[0008] In response to the first size being less than or equal to a first preset value, the target is marked as a first target, and based on the first position, the first target is determined in the (Np)th feature map, and the second information of the first target is also determined; then the second information of the first target is output; where p is a natural number less than N.

[0009] The target detection method provided in this application determines the position and size information of the target based on the Nth feature map obtained by sequential downsampling of N feature maps from a convolutional neural network. Furthermore, when a target smaller than a first preset value is detected in the Nth feature map, the current feature map is determined to be a feature map that cannot meet the target detection accuracy. Therefore, based on the coarse position information of the target in the Nth feature map, it is mapped to the intermediate feature map, i.e., the (Np) feature map, for determination. By detecting targets of different size ratios in different feature maps, the accuracy of the detection results of smaller targets and small targets is improved, thereby improving the accuracy of the position and other information of the detected target obtained from the image to be detected.

[0010] In one possible implementation, the first size indicates the number of pixels in the detection frame of the target.

[0011] One possible implementation further includes, before responding to the first size being less than or equal to the first preset value:

[0012] In response to the first size being greater than the first preset value, first information corresponding to the target is output.

[0013] In one possible implementation, the second information includes a second dimension; the second dimension indicates the size of the first target in the (Np)th feature map;

[0014] The step of determining the first target and its second information in the (Np)th feature map based on the location information of the first target includes:

[0015] Based on the first position, the first target is mapped onto the (Np)th feature map to obtain the first mapping region;

[0016] Based on the first mapping region, a first detection region of the first target is determined in the (Np)th feature map, and based on the first detection region, the second information of the first target is determined; wherein the first detection region is greater than or equal to the first mapping region.

[0017] One possible implementation involves determining a first detection region of the first target in the (Np)th feature map based on the first mapping region, and determining the second information of the first target based on the first detection region, including:

[0018] In the (Np)th feature map, a first detection region containing the first mapping region is determined; wherein the center of the first detection region coincides with the center of the first mapping region.

[0019] Set the feature vector of the region outside the first detection region to the second preset value to obtain the sub-feature map of the (Np)th feature map;

[0020] In the sub-feature map of the (Np)th feature map, the first target and the second information of the first target are determined.

[0021] One possible implementation, wherein the first target is determined based on the first position in the (Np)th feature map, and the second information of the first target is also determined; then the second information of the first target is output, including:

[0022] In response to the second size being greater than a third preset value, second information of the first target is determined to be output; wherein the third preset value is less than the first preset value.

[0023] One possible implementation further includes, in response to the second dimension being greater than a third preset value:

[0024] In response to the second size being less than or equal to the third preset value, the first target corresponding to the second size is marked as the third target;

[0025] Based on the second size of the third target, the third target is mapped from the (Np)th feature map to the (Npq)th feature map to obtain the second mapping region; where q is a natural number less than Np;

[0026] Based on the second mapping region, the third target and the third information of the third target are determined.

[0027] In one possible implementation, before receiving the image to be detected, a training unit is further included to receive a training image; wherein the training image includes a preset target and preset information of the preset target; the training image is input into a convolutional neural network to be trained, and in the Nth feature map of the convolutional neural network to be trained, a detection target and first detection information of the detection target are determined; wherein the first detection information includes a first detection position and a first detection size; in response to the first detection size being less than or equal to the first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map, a second detection information of the detection target is determined, and a first preset target corresponding to the detection target is determined; based on a first error between the second detection information and the preset information of the first preset target, the parameters of the convolutional neural network to be trained are adjusted until the first error is less than a preset error threshold, thereby obtaining the convolutional neural network.

[0028] In one possible implementation, before responding to the first detection size being less than or equal to the first preset value, the method further includes: in response to the first detection size being greater than the first preset value, determining a second preset target corresponding to the detection target among the preset targets;

[0029] The step of adjusting the parameters of the convolutional neural network to be trained based on the first error between the second detection information and the preset information of the first preset target until the first error is less than a preset error threshold, thereby obtaining the convolutional neural network, includes:

[0030] Based on the first error between the second detection information and the preset information of the first preset target, and the second error between the first detection information and the preset information of the second preset target, the parameters of the convolutional neural network to be trained are adjusted until both the first error and the second error are less than a preset error threshold, thereby obtaining the convolutional neural network.

[0031] In one possible implementation, the first error includes the error between the detection frame size of the first detection target and the first preset size, the intersection-over-union ratio between the detection frame of the first detection target and the detection frame of the first preset target, and the offset value between the center of the detection frame of the first detection target and the center of the detection frame of the first preset target; wherein, the first detection target is the detection target whose first detection size is less than or equal to the first preset value.

[0032] In one possible implementation, the second error includes the intersection-union ratio between the detection frame of the second detection target and the detection frame of the second preset target, and the offset value between the center of the detection frame of the second detection target and the center of the detection frame of the second preset target; wherein, the second detection target is the detection target whose first detection size is greater than the first preset value, and the second preset target is a preset target corresponding to the second detection target.

[0033] One possible implementation includes mapping the detection target to the (Np)th feature map based on the first detection position of the detection target, determining the second detection information of the detection target, and determining the first preset target corresponding to the detection target, comprising:

[0034] Based on the first detection location, the detection target is mapped onto the (Np)th feature map, and a first training region containing the detection target and a first target region containing the first training region are determined in the (Np)th feature map; the center of the first target region coincides with the center of the first training region;

[0035] In the first target area, second detection information of the detection target is determined; wherein, the second detection information includes the second detection size;

[0036] In response to the second detection size being less than or equal to the second preset value and greater than the third preset value, the first preset target corresponding to the detection target is determined among the preset targets.

[0037] In one possible implementation, before responding to the second detection size being less than or equal to the second preset value and greater than the third preset value, the method further includes:

[0038] In response to the second detection size being less than or equal to the third preset value, the detection target is mapped to the (Npq)th feature map, and a second training region of the detection target and a second target region containing the second training region are determined in the (Npq)th feature map; wherein the center of the second target region coincides with the center of the second training region, and q is a natural number less than Np;

[0039] In the second target area, third detection information of the detection target is determined, and a third preset target corresponding to the third detection information is determined; the preset information of the third preset target is used to determine the third error corresponding to the detection target.

[0040] Secondly, embodiments of this application provide an apparatus for detecting targets based on a convolutional neural network, comprising:

[0041] First information unit: used to input the image to be detected into the convolutional neural network, and based on the Nth feature map in the convolutional neural network, determine the target in the image to be detected, and the first information of the target; wherein, the number of feature maps in the convolutional neural network is N, and N is a natural number greater than 2; the first information includes a first position and a first size;

[0042] The second information unit is used to mark the target as the first target in response to the first size being less than or equal to the first preset value, and to determine the first target and the second information of the first target in the (Np)th feature map based on the first position; then output the second information of the first target; where p is a natural number less than N.

[0043] In one possible implementation, the device for detecting targets based on convolutional neural networks further includes an output unit, which is configured to output first information corresponding to the target in response to the first size being greater than the first preset value.

[0044] In one possible implementation, the second information includes a second size; the second size indicates the size of the first target in the (Np)th feature map; then the second information unit is specifically used to map the first target to the (Np)th feature map based on the first position to obtain a first mapping region; based on the first mapping region, determine a first detection region of the first target in the (Np)th feature map, and based on the first detection region, determine the second information of the first target; wherein the first detection region is greater than or equal to the first mapping region.

[0045] In one possible implementation, the second information unit is specifically used to determine, in the (Np)th feature map, a first detection region containing the first mapping region; wherein the center of the first detection region coincides with the center of the first mapping region; set the feature vector of the region outside the first detection region to a second preset value to obtain a sub-feature map of the (Np)th feature map; and determine, in the sub-feature map of the (Np)th feature map, the first target and the second information of the first target.

[0046] In one possible implementation, the second information unit is further configured to determine and output second information of the first target in response to the second size being greater than a third preset value; wherein the third preset value is less than the first preset value.

[0047] In one possible implementation, the device for detecting targets based on a convolutional neural network further includes a third information unit, which is configured to, in response to the second size being less than or equal to the third preset value, mark the first target corresponding to the second size as a third target; based on the second size of the third target, map the third target from the (Np)th feature map to the (Npq)th feature map to obtain a second mapping region; wherein q is a natural number less than Np; and based on the second mapping region, determine the third target and the third information of the third target.

[0048] In one possible implementation, the device for detecting targets based on a convolutional neural network further includes a training unit for receiving training images; wherein the training images include a preset target and preset information of the preset target; the training images are input into a convolutional neural network to be trained, and in the Nth feature map of the convolutional neural network to be trained, a detection target and first detection information of the detection target are determined; wherein the first detection information includes a first detection position and a first detection size; in response to the first detection size being less than or equal to the first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map, and second detection information of the detection target is determined, and a first preset target corresponding to the detection target is determined; based on a first error between the second detection information and the preset information of the first preset target, the parameters of the convolutional neural network to be trained are adjusted until the first error is less than a preset error threshold, thereby obtaining the convolutional neural network.

[0049] In one possible implementation, the first error includes the error between the detection frame size of the first detection target and the first preset size, the intersection-over-union ratio between the detection frame of the first detection target and the detection frame of the first preset target, and the offset value between the center of the detection frame of the first detection target and the center of the detection frame of the first preset target; wherein, the first detection target is the detection target whose first detection size is less than or equal to the first preset value.

[0050] In one possible implementation, the device for detecting targets based on convolutional neural networks further includes a first response unit, which is configured to determine a second preset target corresponding to the detected target among the preset targets in response to the first detection size being greater than the first preset value.

[0051] The training unit is specifically used to adjust the parameters of the convolutional neural network to be trained based on the first error between the second detection information and the preset information of the first preset target, and the second error between the first detection information and the preset information of the second preset target, until both the first error and the second error are less than a preset error threshold, thereby obtaining the convolutional neural network.

[0052] In one possible implementation, the training unit is specifically configured to map the detected target onto the (Np)th feature map based on the first detection location, and determine a first training region containing the detected target and a first target region containing the first training region in the (Np)th feature map; the center of the first target region coincides with the center of the first training region; in the first target region, determine second detection information of the detected target; wherein the second detection information includes a second detection size; in response to the second detection size being less than or equal to a second preset value and greater than a third preset value, determine the first preset target corresponding to the detected target among the preset targets.

[0053] In one possible implementation, the device for detecting targets based on a convolutional neural network further includes a second response unit. Specifically, the second response unit is configured to, in response to the second detection size being less than or equal to the third preset value, map the detected target onto a (Npq)-th feature map, and determine a second training region of the detected target and a second target region containing the second training region within the (Npq)-th feature map; wherein the center of the second target region coincides with the center of the second training region, and q is a natural number less than Np; within the second target region, third detection information of the detected target is determined, and a third preset target corresponding to the third detection information is determined; the preset information of the third preset target is used to determine a third error corresponding to the detected target.

[0054] Thirdly, embodiments of this application provide a readable storage medium, including,

[0055] memory,

[0056] The memory is used to store a computer program that, when executed by a processor, causes the apparatus including the readable storage medium to perform the method described in the first aspect and any possible implementation.

[0057] Fourthly, embodiments of this application provide an electronic device, including:

[0058] Memory, used to store computer programs;

[0059] When a processor executes a computer program stored in the memory, it implements the method described in the first aspect and any possible implementation. Attached Figure Description

[0060] Figure 1 A flowchart illustrating a method for detecting targets based on a convolutional neural network, provided in an embodiment of this application;

[0061] Figure 2 A flowchart illustrating a method for training a convolutional neural network to obtain a convolutional neural network, provided in an embodiment of this application;

[0062] Figure 3 This application provides a schematic diagram illustrating the training of a detection head for a mid-layer feature map during the training of a convolutional neural network.

[0063] Figure 4 This is a schematic diagram illustrating a method for detecting an image based on a convolutional neural network, as provided in an embodiment of this application.

[0064] Figure 5 A schematic diagram of the structure of a device for detecting targets based on a convolutional neural network, provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of the structure of an electronic device for detecting targets based on a convolutional neural network, provided in an embodiment of this application. Detailed Implementation

[0066] To address the issue of low target detection accuracy in current convolutional neural networks when processing images containing targets of various sizes, this application provides a method for target detection based on convolutional neural networks. This method also utilizes the convolutional layers in the convolutional neural network to sequentially downsample until the last Nth layer, obtain the Nth feature map, and determine the target. Then, the smaller target is back-mapped to the (Np)th feature map, and the smaller target is detected in the (Np)th feature map to determine the target information of the smaller target. The target information of the smaller target, along with the target information of the larger target detected in the Nth feature map, are output as the target information corresponding to the image to be detected.

[0067] The cascade-based target detection method described above can initially identify smaller targets in the Nth feature map. Then, by determining a feature map with a relatively smaller receptive field in the opposite direction to the downsampling direction, the smaller target is mapped back to this shallower feature map with a smaller receptive field for re-identification. This improves the detection accuracy of small targets, including their location and type, by reducing the error between the receptive field of the small target and its true coordinates. Simultaneously, it effectively avoids the problem of large errors in the output result, particularly in the location information, caused by all targets having equal weights and being identified and determined in the Nth feature map.

[0068] It is worth noting that the aforementioned (Npq)th, (Np)th, and Nth feature maps are labeled sequentially based on the feature maps obtained by downsampling from the convolutional neural network. Correspondingly, the receptive fields of the (Npq)th, (Np)th, and Nth feature maps increase sequentially. Furthermore, the aforementioned Nth feature map is a deep feature map obtained through the convolutional neural network, which is relative to the (Np)th and (Npq)th feature maps. Each feature map can be obtained from one or more modules in the convolutional neural network, and each module can include one or more convolutional layers, fully connected layers, and pooling layers, etc., with flexible configurations.

[0069] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0070] Please refer to Figure 1 This application proposes a method for detecting targets based on convolutional neural networks to improve the accuracy of the detected targets. The method specifically includes the following implementation steps:

[0071] Step 101: Input the image to be detected into the convolutional neural network, and determine the target in the image to be detected and the first information of the target based on the Nth feature map in the convolutional neural network.

[0072] The first information includes a first position and a first size.

[0073] In this convolutional neural network, the number of feature maps is N, where N is a natural number greater than 2. Optionally, the structure of this convolutional neural network can be based on any of the following: R-CNN (Region CNN), Fast R-CNN, Faster R-CNN, or YOLO (You Only Look Once: Unified, Real-Time Object Detection).

[0074] This initial information may include the target's category, location, etc. The category corresponds to the usage scenario and pre-defined labels in the convolutional neural network. For example, it could be a pedestrian.

[0075] The location can be represented using the image coordinate system. For example, it can be the coordinates of the center point of the detection box containing the target, as well as the length and width of the detection box. Alternatively, it can be the coordinates of the top-left vertex of the target's detection box, and the lengths extended from that vertex along the x and y axes; or it can be represented by the coordinates of the top-left vertex, the bottom-left vertex, and the top-right vertex.

[0076] Step 102: In response to the first size being less than or equal to the first preset value, mark the target as the first target, and determine the first target and the second information of the first target in the (Np)th feature map based on the first position; then output the second information of the first target.

[0077] Where p is a natural number less than N. For example, if p = 1, then (Np) is the second to last layer in the convolutional neural network.

[0078] This first preset value is also used to indicate the size of the target, and is used to distinguish and filter out small and / or relatively small targets that are difficult to identify with high precision in the Nth layer feature map. Specifically, when the first size is less than or equal to the first preset value, the target is marked as the first target and mapped to the (Np)th feature map for fine detection in the (Np)th feature map.

[0079] When the first size is greater than the first preset value, it indicates that the target is a large target. Therefore, the detection accuracy of the target meets the preset accuracy threshold, and the target can be marked as the second target, and the second information indicating the second target can be output.

[0080] Similarly, this second information includes the type information and location information of the second target. The type information depends on the specific use case and training labels; for example, it could be a car, a truck, or a pedestrian.

[0081] Furthermore, the following describes in detail an implementation method for mapping a first target to the feature map of an (Np)-layer convolutional layer to determine the second information of the first target:

[0082] First, based on the first position, the second target is mapped from the Nth feature map to the (Np)th feature map, resulting in the first mapped region. This first mapped region is the possible region of the first target in the (Np)th feature map. In essence, the detection box of the second target is mapped to the corresponding position on the (Np)th feature map.

[0083] Then, within the first mapping region, the first detection region of the first target is determined in the (Np)th feature map. The first detection region is then detected using the detection head of the (Np)th feature map in the convolutional neural network to determine the second information of the first target.

[0084] The second information is the target information obtained based on the first target in the (N - p)-th feature map.

[0085] It can be understood that the first target itself is small in size, and in the N-th feature map, the receptive field of any target includes richer feature information. Therefore, when the first target is mapped back to the (N - p)-th feature map, the first target may be located at a position close to the center within the first mapping region, or may be located close to the edge of the first mapping region.

[0086] Thus, in some embodiments, the first mapping region is used as the first detection region for direct detection. In other embodiments, to avoid the influence of the first target being close to the edge of the first mapping region on the detection accuracy, a first detection region slightly larger than the first mapping region can be determined. The center of the first detection region coincides with the center of the first mapping region. For example, the first mapping region with the same shape and aspect ratio as the detection frame is extended 2, 3, or 5 pixel points in each direction around the detection frame. That is, the four sides of the first mapping region are translated along the positive and negative x-axes and the positive and negative y-axes, and the center of the translated first detection region is the center of the first mapping region, forming a "hui" (Chinese character for "return") - shaped structure.

[0087] Next, after determining the first detection region, in the (N - p)-th feature map, all feature vectors in the region outside the first detection region are uniformly set to a second preset value, for example, 0, to ensure that the second target in the first detection region can be detected during the second target recognition process without being affected by unnecessary interference, and a sub - feature map of the (N - p)-th feature map is obtained.

[0088] Finally, in the sub - feature map of the (N - p)-th feature map, the first target and the second information of the first target can be determined.

[0089] Furthermore, to avoid the problem that targets smaller than the first target cannot be detected with high precision in the sub - feature map of the (N - p)-th feature map, in some embodiments, after determining the second information of the first target in the (N - p)-th feature map, further judgment is made according to the second position and the second size of the second information. The following is a specific description:

[0090] In response to the second size being greater than a third preset value, it is determined that in the current sub - feature map of the (N - p)-th feature map, the receptive field of the first target, that is, the second target, has a low error with its true coordinates and meets the accuracy threshold, then the second information of the first target is output. Or,

[0091] In response to the second size being less than or equal to the aforementioned third preset value, it is determined that because the size of the first target, i.e., the second target, is small, the detection accuracy of the sub-feature map of the current (Np)th feature map and its detection head for detecting the second target can no longer meet the aforementioned accuracy threshold. Therefore, the first target is mapped from the (Np)th feature map in the convolutional neural network to the (Npq)th feature map to obtain the second mapping region. Here, q is a natural number less than Np. q can be the same as p, for example, both can be 1. q and p can also be different.

[0092] Then, based on the second mapping region, the third target and the third information of the third target in the (Npq) feature map can be determined.

[0093] The determination of the third target in the (Npq)th feature map is consistent with the method for determining the second target in the (Np)th convolutional layer feature map. Similarly, a detection head corresponding to the (Npq)th feature map is set up to identify and determine the third target and its third information. For example, the second mapping region can be directly used as the second detection region, and detection can be performed in the second detection region of the (Npq)th feature map. Alternatively, detection can be performed in a second detection region that coincides with the center of the second mapping region and is slightly larger than the second mapping region to determine the third information of the third target. Furthermore, to avoid unnecessary feature interference, the feature vectors of regions outside the second detection region can still be set to the aforementioned second preset value before detection.

[0094] Unless otherwise specified, in the embodiments of this application, each size, such as the first size and the second size, can uniformly indicate the number of pixels occupied by the corresponding target, and can also uniformly indicate the number of pixels occupied by the corresponding target detection box.

[0095] The first preset value used to distinguish between large and small targets, and the third preset value used to further divide small targets into medium and small targets, can be defined and set with reference to the COCO (Common Objects in Context) dataset, a general dataset in the field of object detection.

[0096] Assuming a convolutional neural network contains 5 convolutional layers, a 32*32*3 region in the input image will form a 1*1*d feature image after convolutional downsampling. The "3" and "d" refer to feature channels. The first preset value could be, for example, 96*96 pixels. The second preset value could be 32*32 pixels. Therefore, the first size of the second target, i.e., the large target, is greater than 96*96 pixels, while the size of the first target is less than or equal to 96*96 pixels. The first target, which can be classified as a medium target and identified in the medium-layer feature map (i.e., the (Np)th feature map), has a size of 32*32 - 96*96 pixels. Therefore, the third target, with the smallest first size and requiring identification in the shallow feature map, has a size less than or equal to 32*32 pixels.

[0097] The first and second preset values ​​vary with different scales and receptive field feature maps. However, since the first and second preset values ​​are determined by converting the size of each target in each feature map to the initial image, i.e. the image to be detected, the first and second preset values ​​are uniform values ​​in the same feature map and the image to be detected.

[0098] Furthermore, the following provides a detailed explanation of the aforementioned convolutional neural network and the method for training detection heads corresponding to feature maps of different convolutional layers for detecting different size ranges. Please refer to [link / reference needed]. Figure 2 .

[0099] Step 201: Receive training images.

[0100] The training images include a preset target and preset information about the preset target.

[0101] Step 202: Input the training image into the convolutional neural network to be trained, and determine the detection target and the first detection information of the detection target in the Nth feature map of the convolutional neural network to be trained.

[0102] The first detection information includes the first detection position and the first detection size.

[0103] Step 203: In response to the first detection size being less than or equal to the first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map of the convolutional neural network to be trained, the second detection information of the detection target is determined, and the first preset target corresponding to the detection target is determined.

[0104] Alternatively, in some embodiments, in response to the first detection size being greater than a first preset value, and the detection target being determined to be a large target, a second preset target corresponding to the detection target is determined from the preset targets. Based on this, a second error can be determined between the first detection information of the detection target and the preset information of the second preset target. Thus, when adjusting the parameters of the convolutional neural network to be trained, the parameters of the convolutional neural network to be trained can be adjusted based on the first error between the second detection information and the preset information of the first preset target, and the second error between the first detection information and the preset information of the second preset target, until both the first error and the second error are less than a preset error threshold, thereby obtaining the convolutional neural network.

[0105] The first preset goal is the goal included in the preset goals.

[0106] Specifically, since the convolutional neural network to be trained will continue to use the same methods for target recognition and target information determination during training, the detected target will be mapped to the (Np)th feature map to determine the implementation method of the second detection information. This is consistent with the mapping of the second and third targets and the determination of target information in step 102.

[0107] Similarly, based on the first detection location, the detection target is first mapped to the (Np)th feature map, and a first training region containing the detection target and a first target region containing the first training region are determined in the (Np)th feature map; the center of the first target region coincides with the center of the first training region. Likewise, the first target region may coincide with the first training region or may be slightly larger than the first training region.

[0108] Then, within the first training region, second detection information of the target is determined. The second detection information includes the second detection size.

[0109] Next, in response to the second detection size in the second detection information being less than or equal to the second preset value and greater than the third preset value, the first preset target corresponding to the detection target is determined among the preset targets.

[0110] Alternatively, in response to the second detection size being less than or equal to the third preset value, the detected target can be determined to be a small target. Therefore, the detected target is first mapped to the (Npq)th feature map to train the detection head corresponding to the (Npq)th feature map: a second training region for the detected target and a second target region containing the second training region are determined in the (Npq)th feature map; wherein the center of the second target region coincides with the center of the second training region, and q is a natural number less than Np;

[0111] Then, in the second target area, the third detection information of the detection target is determined, and the third preset target corresponding to the third detection information is determined; the preset information of the third preset target is used to determine the third error corresponding to the detection target.

[0112] Step 204: Based on the first error between the second detection information and the preset information of the first preset target, adjust the parameters of the convolutional neural network to be trained until the first error is less than a preset error threshold, and obtain the convolutional neural network.

[0113] Specifically, the first error includes the error between the detection box size of the first detected target and a first preset value. This error can be used as a penalty for the first detected target exceeding the limit, and can be calculated using h1*w1 / 96*96. Here, h1 is the height of the detection box of the first detected target in the (Np)th feature map converted to the height of the detection box in the image to be detected; w1 is the width of the detection box of the first detected target in the (Np)th feature map converted to the width of the detection box in the image to be detected.

[0114] The first detection target is a detection target whose first detection size is less than or equal to the first preset value.

[0115] The first error also includes the intersection-union ratio (IUGR) between the detection bounding box of the first detected target and the detection bounding box of the first preset target, the IUGR between the two corresponding detection bounding boxes, and the offset of the detection bounding box center coordinates. This error can be calculated using the following formula:

[0116] Among them, R truth R_pred is the detection box in the image to be detected obtained based on the preset information (i.e., preset size) of the first preset target. R_pred is the detection box of the first target in the (Np)th feature map converted to the detection box in the image to be detected. c_truth is the center point coordinate of the detection box in the image to be detected obtained based on the preset information (i.e., preset size) of the first preset target. c_pred is the center point coordinate of the detection box of the first target in the (Np)th feature map converted to the detection box in the image to be detected. ρ is the Euclidean distance between the center points of the two detection boxes. a is the diagonal distance of the detection box in the image to be detected obtained based on the preset information (i.e., preset size) of the first preset target.

[0117] The detection target corresponding to the second error has a different detection size than the first detection target and can be labeled as the second detection target. Since the first detection size of the second detection target is larger than the first preset value, this second detection target is a large target and does not exceed the size limit. Therefore, the second error includes the intersection-union ratio (IUU) between the detection frame of the second detection target and the detection frame of the second preset target, and the offset between the center coordinates of the detection frame of the second detection target and the center coordinates of the detection frame of the second preset target. The calculation of this second error is also based on... get.

[0118] Among them, R truth The detection box in the image to be detected is obtained based on the preset information (i.e., preset size) of the second preset target. The detection box in the Nth feature map is transformed into the detection box in the image to be detected. The center point coordinates of the detection box in the image to be detected are obtained based on the preset information (i.e., preset size) of the second preset target. The center point coordinates of the detection box in the Nth feature map are transformed into the detection box in the image to be detected. The Euclidean distance between the center points of the two detection boxes is ρ. The diagonal distance of the detection box in the image to be detected is obtained based on the preset information (i.e., preset size) of the second preset target.

[0119] For targets with smaller detection sizes, after detection on the (Np)th feature map, based on the detected size information, it can be further mapped forward to the (Npq)th feature map for detection. The error is then determined based on the detection information in the (Npq)th feature map, and the parameters of the convolutional neural network to be trained are adjusted to improve the detection accuracy of the detection head on the (Npq)th feature map. The determination and handling of this error are consistent with the handling of the first error.

[0120] The following example illustrates how, after detecting a target whose first detection size is less than or equal to a first preset value using the Nth feature map, the target is mapped to the (N-1)th layer for detection and training. Please refer to [link / reference]. Figure 3 .

[0121] like Figure 3 As shown, the deep layer corresponds to the Nth feature map, the middle layer corresponds to the (N-1)th feature map, and the shallow layer corresponds to the (N-2)th feature map. The large target is the aforementioned second detection target, the medium target is the aforementioned first detection target, and the small target can be the third detection target.

[0122] Specifically, each of the deep, middle, and shallow feature maps is equipped with a corresponding detection head. When the detection head of the deep feature map detects a target smaller than a first preset value, the target can be identified as a medium-sized target and / or a small target. Furthermore, the position detected by the detection head of the deep feature map is the coarse position of the aforementioned medium-sized target and / or small target.

[0123] Based on the coarse location of the aforementioned medium and / or small targets, the feature map of the middle layer (convolutional layer) can be updated. Feature vectors outside the coarse location region, or regions containing the coarse location and whose detection boxes are appropriately enlarged proportionally, are set to 0. Feature vectors within the region are not processed. Therefore, the detection head based on the middle layer feature map can perform detection on the updated feature map. This essentially involves detecting regions where the feature vectors are not 0, and after detection, determining whether a target is a medium target based on the obtained detection information.

[0124] Once the target is identified as a mid-level target, a regression loss function is determined based on the aforementioned error information, and a classification loss function is determined based on the type information and preset information in the detection information, so as to adjust the detection head of the mid-level feature map in reverse.

[0125] Conversely, if the target is not a medium target based on the detection information obtained by the detection head of the middle-layer feature map on the updated feature map, then the position information in this detection information can be used as the coarse position of the small target. Therefore, the feature map of the small target can be further updated and detected based on the coarse position of the small target, thereby achieving the purpose of training the detection head of the shallow feature map. The specific implementation steps are the same as the training method of the detection head of the middle-layer feature map mentioned above, and will not be repeated here.

[0126] Furthermore, according to the above method, when the detection errors obtained by the detection heads of the deep feature maps, mid-level feature maps, and shallow feature maps of the aforementioned convolutional neural network to be trained are all less than a preset error threshold, the convolutional neural network is determined to be obtained. The following example illustrates the method of using the aforementioned convolutional neural network for detection after receiving the image to be detected. Please refer to [link / reference]. Figure 4 .

[0127] First, an image is input. Then, the backbone network of a convolutional neural network is used to downsample and extract features, thereby obtaining the corresponding feature maps.

[0128] Next, a deep feature map detection head is used to detect the deep feature map, obtaining detection information for large targets, including positional information, and coarse positional information for small and medium-sized targets. Then, based on the coarse positional information of the small and medium-sized targets, a mid-level feature map detection head is used to detect the mid-level feature map, obtaining detection information for medium-sized targets, which includes positional information and coarse positional information for small targets.

[0129] Next, using the coarse location information of the small target, the detection head of the shallow feature map is used to detect the shallow feature map to obtain the detection information of the small target, which includes the location information of the small target.

[0130] Finally, the detection information of large, medium, and small targets can be combined and output as the detection result of the image to be detected.

[0131] It is worth noting that when determining the detection information of small and medium targets in the deep feature map or the detection information of small targets in the medium feature map, all detection information, including category information and location information, can be detected. Alternatively, to improve efficiency, only the location information can be detected and used as a coarse location for mapping.

[0132] Based on the same inventive concept, this application provides a device for detecting targets based on a convolutional neural network, which is similar to the aforementioned device. Figure 2 The method for detecting targets based on convolutional neural networks shown corresponds to this device. For a detailed description of the implementation method, please refer to the foregoing description in the method embodiments section. Repeated descriptions will not be repeated here. Figure 5 The device includes:

[0133] First information unit 501: used to input the image to be detected into the convolutional neural network, and based on the Nth feature map in the convolutional neural network, determine the target in the image to be detected, and the first information of the target.

[0134] The number of feature maps in the convolutional neural network is N, where N is a natural number greater than 2; the first information includes a first position and a first size.

[0135] The device for detecting targets based on convolutional neural networks further includes an output unit, which is used to output first information corresponding to the target in response to the first size being greater than the first preset value.

[0136] The second information unit 502 is used to mark the target as a first target in response to the first size being less than or equal to a first preset value, and to determine the first target and its second information in the (Np)th feature map based on the first position; then output the second information of the first target.

[0137] Where p is a natural number less than N.

[0138] Further, the aforementioned second information includes a second size; the second size indicates the size of the first target in the (Np)th feature map; then the second information unit 502 is specifically used to map the first target to the (Np)th feature map based on the first position to obtain a first mapping region; based on the first mapping region, determine a first detection region of the first target in the (Np)th feature map, and based on the first detection region, determine the second information of the first target; wherein, the first detection region is greater than or equal to the first mapping region.

[0139] The second information unit 502 is specifically used to determine the first detection region containing the first mapping region in the (Np)th feature map; wherein the center of the first detection region coincides with the center of the first mapping region; set the feature vector of the region outside the first detection region to a second preset value to obtain a sub-feature map of the (Np)th feature map; and determine the first target and the second information of the first target in the sub-feature map of the (Np)th feature map.

[0140] The second information unit 502 is further configured to determine and output second information of the first target in response to the second size being greater than a third preset value; wherein the third preset value is less than the first preset value.

[0141] The device for detecting targets based on a convolutional neural network further includes a third information unit, which is used to mark the first target corresponding to the second size as a third target in response to the second size being less than or equal to the third preset value; based on the second size of the third target, map the third target from the (Np)th feature map to the (Npq)th feature map to obtain a second mapping region; where q is a natural number less than Np; and based on the second mapping region, determine the third target and the third information of the third target.

[0142] The apparatus for detecting targets based on a convolutional neural network further includes a training unit, which receives a training image; wherein the training image includes a preset target and preset information of the preset target; the training image is input into a convolutional neural network to be trained, and in the Nth feature map of the convolutional neural network to be trained, a detection target and first detection information of the detection target are determined; wherein the first detection information includes a first detection position and a first detection size; in response to the first detection size being less than or equal to the first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map, and second detection information of the detection target is determined, and a first preset target corresponding to the detection target is determined; based on a first error between the second detection information and the preset information of the first preset target, the parameters of the convolutional neural network to be trained are adjusted until the first error is less than a preset error threshold, thereby obtaining the convolutional neural network.

[0143] The first error includes the error between the detection frame size of the first detection target and the first preset size, the intersection-over-union ratio between the detection frame of the first detection target and the detection frame of the first preset target, and the offset value between the center of the detection frame of the first detection target and the center of the detection frame of the first preset target; wherein, the first detection target is the detection target whose first detection size is less than or equal to the first preset value.

[0144] The device for detecting targets based on convolutional neural networks further includes a first response unit, which is used to determine a second preset target corresponding to the detected target in response to the first detection size being greater than the first preset value.

[0145] The training unit is specifically used to adjust the parameters of the convolutional neural network to be trained based on the first error between the second detection information and the preset information of the first preset target, and the second error between the first detection information and the preset information of the second preset target, until both the first error and the second error are less than a preset error threshold, thereby obtaining the convolutional neural network.

[0146] The training unit is specifically configured to map the detected target to the (Np)th feature map based on the first detection location, and determine a first training region containing the detected target and a first target region containing the first training region in the (Np)th feature map; the center of the first target region coincides with the center of the first training region; in the first target region, determine second detection information of the detected target; wherein the second detection information includes the second detection size; in response to the second detection size being less than or equal to the second preset value and greater than the third preset value, determine the first preset target corresponding to the detected target among the preset targets.

[0147] The device for detecting targets based on convolutional neural networks further includes a second response unit. Specifically, the second response unit is configured to, in response to the second detection size being less than or equal to the third preset value, map the detected target onto a (Npq)th feature map, and determine a second training region for the detected target and a second target region containing the second training region within the (Npq)th feature map; wherein the center of the second target region coincides with the center of the second training region, and q is a natural number less than Np; within the second target region, third detection information of the detected target is determined, and a third preset target corresponding to the third detection information is determined; the preset information of the third preset target is used to determine a third error corresponding to the detected target.

[0148] Based on the same inventive concept, embodiments of this application also provide a readable storage medium, including:

[0149] memory,

[0150] The memory is used to store a computer program that, when executed by a processor, causes the apparatus including the readable storage medium to perform the method for detecting targets based on a convolutional neural network as described above.

[0151] Based on the same inventive concept as the aforementioned method for detecting targets using convolutional neural networks, this application also provides an electronic device that can implement the functions of the aforementioned method for detecting targets using convolutional neural networks. Please refer to [link / reference]. Figure 6 The electronic device includes:

[0152] At least one processor 601 and a memory 602 connected to at least one processor 601. In this embodiment, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 The example shown is the connection between processor 601 and memory 602 via bus 600. Bus 600 is... Figure 6The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 600 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 6 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller; there is no restriction on the name.

[0153] In this embodiment, memory 602 stores instructions executable by at least one processor 601. By executing the instructions stored in memory 602, at least one processor 601 can perform the aforementioned method for detecting targets based on a convolutional neural network. Processor 601 can implement... Figure 5 The functions of each module in the device shown.

[0154] The processor 601 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 602 and calling data stored in memory 602, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0155] In one possible design, processor 601 may include one or more processing units. Processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0156] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method for detecting targets based on convolutional neural networks disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0157] Memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 602 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 602 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0158] By designing and programming the processor 601, the code corresponding to the convolutional neural network-based target detection method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute it during runtime. Figure 1 The steps of the method for detecting targets based on a convolutional neural network are shown. How to design and program the processor 601 is a technique well-known to those skilled in the art and will not be described further here.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0160] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0164] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting targets based on convolutional neural networks, characterized in that, include: Receive training images; wherein the training images include a preset target and preset information of the preset target; The training image is input into the convolutional neural network to be trained, and the detection target and the first detection information of the detection target are determined in the Nth feature map of the convolutional neural network to be trained; wherein, the first detection information includes a first detection position and a first detection size; In response to the first detection size being less than or equal to a first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map, the second detection information of the detection target is determined, and the first preset target corresponding to the detection target is determined; Based on the first error between the second detection information and the preset information of the first preset target, the parameters of the convolutional neural network to be trained are adjusted until the first error is less than a preset error threshold, thereby obtaining the convolutional neural network. The image to be detected is input into the convolutional neural network, and the target in the image to be detected and the first information of the target are determined based on the Nth feature map in the convolutional neural network; wherein, the number of feature maps in the convolutional neural network is N, and N is a natural number greater than 2; the first information includes a first position and a first size; In response to the first size being less than or equal to a first preset value, the target is marked as a first target, and based on the first position, the first target is determined in the (Np)th feature map, and the second information of the first target is also determined; then the second information of the first target is output; where p is a natural number less than N.

2. The method as described in claim 1, characterized in that, The response before the first size is less than or equal to the first preset value further includes: In response to the first size being greater than the first preset value, first information corresponding to the target is output.

3. The method as described in claim 1, characterized in that, The second information includes a second size; the second size indicates the size of the first target in the (Np)th feature map; The step of determining the first target and its second information in the (Np)th feature map based on the location information of the first target includes: Based on the first position, the first target is mapped onto the (Np)th feature map to obtain the first mapping region; Based on the first mapping region, a first detection region of the first target is determined in the (Np)th feature map, and based on the first detection region, the second information of the first target is determined; wherein the first detection region is greater than or equal to the first mapping region.

4. The method as described in claim 3, characterized in that, Based on the first mapping region, a first detection region of the first target is determined in the (Np)th feature map, and based on the first detection region, the second information of the first target is determined, including: In the (Np)th feature map, a first detection region containing the first mapping region is determined; wherein the center of the first detection region coincides with the center of the first mapping region. Set the feature vector of the region outside the first detection region to the second preset value to obtain the sub-feature map of the (Np)th feature map; In the sub-feature map of the (Np)th feature map, the first target and the second information of the first target are determined.

5. The method as described in claim 4, characterized in that, Based on the first position, the first target is determined in the (Np)th feature map, and the second information of the first target is obtained; then the second information of the first target is output, including: In response to the second size being greater than a third preset value, second information of the first target is determined to be output; wherein the third preset value is less than the first preset value.

6. The method as described in claim 5, characterized in that, The response before the second size is greater than the third preset value further includes: In response to the second size being less than or equal to the third preset value, the first target corresponding to the second size is marked as the third target; Based on the second size of the third target, the third target is mapped from the (Np)th feature map to the (Npq)th feature map to obtain the second mapping region; where q is a natural number less than Np; Based on the second mapping region, the third target and the third information of the third target are determined.

7. The method as described in claim 6, characterized in that, The first error includes the error between the detection frame size of the first detection target and the first preset value, the intersection-over-union ratio between the detection frame of the first detection target and the detection frame of the first preset target, and the offset value between the center of the detection frame of the first detection target and the center of the detection frame of the first preset target; wherein, the first detection target is the detection target whose first detection size is less than or equal to the first preset value.

8. The method as described in claim 6, characterized in that, The step of mapping the detection target to the (Np)th feature map based on the first detection position of the detection target, determining the second detection information of the detection target, and determining the first preset target corresponding to the detection target includes: Based on the first detection location, the detected target is mapped to the (Np)th feature map, and a first training region containing the detected target and a first target region containing the first training region are determined in the (Np)th feature map; the center of the first target region coincides with the center of the first training region; In the first target area, second detection information of the detection target is determined; wherein, the second detection information includes a second detection size; In response to the second detection size being less than or equal to the second preset value and greater than the third preset value, the first preset target corresponding to the detection target is determined among the preset targets.

9. The method as described in claim 8, characterized in that, Before responding to the second detection size being less than or equal to the second preset value and greater than the third preset value, the method further includes: In response to the second detection size being less than or equal to the third preset value, the detection target is mapped to the (Npq)th feature map, and a second training region of the detection target and a second target region containing the second training region are determined in the (Npq)th feature map; wherein the center of the second target region coincides with the center of the second training region, and q is a natural number less than Np; In the second target area, third detection information of the detection target is determined, and a third preset target corresponding to the third detection information is determined; the preset information of the third preset target is used to determine the third error corresponding to the detection target.

10. A device for detecting targets based on a convolutional neural network, characterized in that, include: First information unit: Used to receive a training image; wherein the training image includes a preset target and preset information of the preset target; The training image is input into a convolutional neural network to be trained, and in the Nth feature map of the convolutional neural network to be trained, a detection target and first detection information of the detection target are determined; wherein the first detection information includes a first detection position and a first detection size; In response to the first detection size being less than or equal to a first preset value, based on the first detection position of the detection target, the detection target is mapped to the (Np)th feature map, and second detection information of the detection target is determined, and a first preset target corresponding to the detection target is determined; Based on a first error between the second detection information and the preset information of the first preset target, the parameters of the convolutional neural network to be trained are adjusted until the first error is less than a preset error threshold, thus obtaining the convolutional neural network; The image to be detected is input into the convolutional neural network, and based on the Nth feature map in the convolutional neural network, a target in the image to be detected and first information of the target are determined; wherein the number of feature maps in the convolutional neural network is N, where N is a natural number greater than 2; the first information includes a first position and a first size; The second information unit is used to mark the target as the first target in response to the first size being less than or equal to the first preset value, and to determine the first target and the second information of the first target in the (Np)th feature map based on the first position; then output the second information of the first target; where p is a natural number less than N.

11. A readable storage medium, characterized in that, include, memory, The memory is used to store a computer program that, when executed by a processor, causes the apparatus including the readable storage medium to perform the method as described in any one of claims 1-9.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method as described in any one of claims 1-9.

Images