Image processing method, image processing device and computer readable storage medium

Abstract

An embodiment of the present invention provides an image processing method, an apparatus, and a computer-readable storage medium, wherein the image processing method comprises: obtaining an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information according to the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is less than the first number.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, an image processing device and a computer-readable storage medium. Background Art

[0002] Currently, object detection technology is one of the core issues of computer vision. It is widely used in the fields of product defect detection in industrial production, autonomous driving, video surveillance, etc., and is used to provide extraction and detection of objects of interest in images or videos. Object detection technology can not only determine the specific category of the object to be detected in a detection image (for example, whether it belongs to a pedestrian, vehicle or animal, etc.), but also provide the specific position of each object to be detected in the detection image (for example, generate a predicted bounding box surrounding each object to be detected).

[0003] Generally, target detection technology can be classified into anchor-based target detection technology or anchor-free target detection technology. However, in anchor-based target detection technology, when training a neural network or actually performing target detection, multiple anchor frames are usually first generated uniformly in advance according to a fixed size and aspect ratio on the feature map of the input training image or detection image, so as to match the real bounding box of the training object when training the neural network or generate the predicted bounding box of the object to be detected when performing target detection based on these anchor frames. However, these pre-generated anchor frames are usually very dense, while the training objects or objects to be detected are relatively sparse in comparison, which results in a large amount of time and computation being wasted on processing invalid anchor frames. In addition, the position, size and aspect ratio of the pre-generated anchor frames are fixed, while the specific size and position of the object are extremely variable, which makes it difficult to pre-generate suitable anchor frames for various object situations, thereby making the neural network less robust to object changes. Moreover, during the training process, the anchor box is usually only responsible for regressing the real bounding box of the object whose intersection over union (IOU) meets the threshold requirement and has the maximum IOU, which may result in other objects that meet the threshold requirement of IOU with the anchor box but do not have the maximum IOU and therefore cannot participate in the training. Although the anchor-free object detection technology does not generate anchor boxes but directly regresses the location information of the object, it also usually faces the problem of low recall rate.

[0004] Therefore, there is a need for an image processing method and device that can better filter, regress and generate anchor frames, so as to improve the efficiency of training neural networks and performing target detection, and enhance the robustness of neural networks against object changes. Summary of the invention

[0005] To solve the above technical problem, according to one aspect of the present invention, there is provided an image processing method, comprising: acquiring an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information according to the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is smaller than the first number.

[0006] According to another aspect of the present invention, there is provided an image processing device, comprising: an acquisition unit, configured to acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; a semantic information generation unit, configured to generate semantic information according to the image to be processed; and an anchor frame filtering unit, configured to filter the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is smaller than the first number.

[0007] According to another aspect of the present invention, there is provided an image processing device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor performs the following steps: acquiring an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information according to the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is smaller than the first number.

[0008] According to yet another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information according to the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is smaller than the first number.

[0009] According to the above-mentioned image processing method, device and computer-readable storage medium of the present invention, a large number of pre-generated anchor frames can be filtered according to semantic information to remove invalid anchor frames therein, so that the filtered anchor frames are further used to train neural networks and perform target detection. Such an image processing method, device and computer-readable storage medium can reduce the processing of invalid anchor frames in the training process or target detection process, and improve the efficiency of training neural networks and performing target detection.

[0010] In addition, according to the above-mentioned image processing method, device and computer-readable storage medium of the present invention, the training objects can be divided into regular samples for training the neural network to adjust the parameters for anchor box regression, and difficult samples for training the neural network to adjust the parameters for anchor box generation, thereby enhancing the robustness of the neural network against object changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0012] Figure 1 A flowchart showing an image processing method according to an embodiment of the present invention is shown;

[0013] Figure 2 An example of generating semantic information of an image to be processed according to an embodiment of the present invention is shown;

[0014] Figure 3 A flow chart of classifying training objects using filtered anchor boxes and true bounding boxes according to one embodiment of the present invention is shown;

[0015] Figure 4 An example of a training object that cannot be matched with a filtered anchor box according to an embodiment of the present invention is shown;

[0016] Figure 5 A schematic diagram showing a prediction result output according to parameters for anchor box generation according to one embodiment of the present invention;

[0017] Figure 6 A block diagram of an image processing apparatus 600 according to an embodiment of the present invention is shown;

[0018] Figure 7 FIG. 7 is a block diagram of an image processing apparatus 700 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will describe an image processing method, device and computer-readable storage medium according to an embodiment of the present invention with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0020] The following will refer to Figure 1 The image processing method according to the embodiment of the present invention is described. The image processing method according to the embodiment of the present invention can be applied to static images as well as video frames in a video that changes over time, etc., and is not limited here. Figure 1 A flow chart of the image processing method 100 is shown.

[0021] like Figure 1 As shown, in step S101, an image to be processed and a plurality of pre-generated anchor frames on the image to be processed are obtained, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number.

[0022] As is well known to those skilled in the art, the process of using a neural network for target detection can generally be divided into two specific processes, namely, a "training process" and a "target detection process". The "training process" refers to training a neural network using a pre-labeled training image to adjust the parameters in the neural network until the loss function converges; and the "target detection process" refers to sending the detection image to the trained neural network after the training process is completed to output the target detection result predicted according to the adjusted parameters. Accordingly, depending on whether step S101 is specifically a step for a training process or a step for a target detection process, the image to be processed may be a training image for training a neural network, or a detection image for target detection. When the image to be processed is a training image, at least one object contained in the image to be processed is a training object, and the training object is pre-labeled with a training object category and a real bounding box surrounding the training object to indicate its specific position on the image to be processed; when the image to be processed is a detection image, the image to be processed may contain at least one object to be detected, and the target detection result may be a predicted class of the object to be detected output on the image to be processed and a predicted bounding box surrounding the object to be detected to indicate its specific position on the image to be processed.

[0023] In this step, multiple pre-generated anchor frames on the image to be processed as a training image or a detection image can be obtained, and these pre-generated anchor frames are used together with the real bounding box of the training object to train the neural network in the case of training, or, in the case of target detection, these pre-generated anchor frames are sent to the trained neural network to obtain the final predicted bounding box of the object to be detected. Optionally, during the training process or the target detection process, rectangular frames with different positions, sizes and aspect ratios are generated as the multiple pre-generated anchor frames with each pixel or part of the pixel on the image to be processed as the center. In one example, the position, size and aspect ratio of the pre-generated anchor frame can be manually set. In another example, an anchor frame that is more in line with the training object or the object to be detected can be automatically generated by a k-means clustering algorithm or the like.

[0024] For ease of explanation, the number of anchor frames pre-generated in this step is referred to as the first number. Generally, in order to better cover and match real bounding boxes with various sizes and positions or obtain predicted bounding boxes of objects to be detected with different positions, sizes and shapes, the pre-generated anchor frames are often very dense and large in number. In contrast, the training objects contained in the training images or the objects to be detected contained in the detection images are usually sparse, so the number of anchor frames that can be matched with the training objects and actually participate in the training during the training process, or the number of anchor frames through which the final predicted bounding boxes can be generated during the target detection process is actually very limited, which results in a large amount of time and calculation being wasted in the training process or the target detection process to process invalid anchor frames, thereby affecting the efficiency of training the neural network or performing target detection. In an embodiment of the present invention, the number of invalid anchor frames can be reduced by filtering the pre-generated anchor frames according to the semantic information described in detail below, thereby improving the efficiency of training the neural network or performing target detection.

[0025] In step S102, semantic information is generated according to the image to be processed.

[0026] In this step, semantic information is first generated according to the image to be processed, so that the pre-generated anchor frames can be filtered according to the semantic information in a subsequent process. In one example, generating semantic information according to the image to be processed includes: obtaining a pixel category of each pixel on the image to be processed according to the image to be processed.

[0027] Figure 2 An example of generating semantic information of an image to be processed according to an embodiment of the present invention is shown. Figure 2As shown in the figure, the semantic information of the image to be processed on the left is predicted, and the prediction result shown in the image on the right is obtained, wherein each pixel on the image to be processed is predicted to have a pixel category of "person", "car" or "background". Optionally, the pixel category of each pixel on the image to be processed can also correspond to a semantic segmentation area on the image to be processed. For example, Figure 2 As shown, in the image on the right, different colors are used to represent the semantic segmentation areas corresponding to the pixel categories "person", "car" or "background", where the semantic segmentation areas corresponding to various pixel categories can be composed of one or more interconnected or closed sub-areas. In one example, the semantic information can be predicted directly for the image to be processed. In another example, the grayscale image of the image to be processed can be obtained first, and then the semantic information can be predicted for the grayscale image of the image to be processed, which is not limited here.

[0028] The above-mentioned method of predicting each pixel on the image to be processed as having a pixel category such as "person", "car" or "background" is only an example. In practical applications, the pixel category of each pixel on the image to be processed can be roughly predicted as "foreground" or "background" so as not to generate pixel categories involving specific object categories, or the pixel category of each pixel on the image to be processed can be more carefully predicted and divided and a more specific pixel category can be generated, so that the pixel category is adapted to the category of the training object or detection object that the user is concerned about, so as to meet different anchor box filtering requirements.

[0029] In step S103, the anchor boxes are filtered using the semantic information to obtain filtered anchor boxes, wherein the number of the filtered anchor boxes is a second number, which is smaller than the first number.

[0030] In this step, when the pixel category of each pixel on the image to be processed is obtained, the pre-generated anchor frame can be filtered based on the pixel category, and part of the anchor frame is removed according to the pixel category to obtain the filtered anchor frame.

[0031] In one example, the anchor frames can be filtered according to the pixel categories of the pixels at the center positions of the anchor frames. For example, when the pixel category of the pixels at the center positions of the anchor frames belongs to "background", it can be considered that the offset of the anchor frame from the real bounding box of the training object pre-annotated in the training process or the predicted bounding box of the object to be detected finally generated in the target detection process is large, and accordingly belongs to an invalid anchor frame in the training process or the target detection process. Therefore, in this example, the anchor frames can be filtered according to the pixel categories of the pixels at the center positions of the anchor frames, so as to remove the anchor frames whose pixel categories of the pixels at the center positions belong to the background to obtain the filtered anchor frames. The number of filtered anchor frames obtained in this step is called the second number, and since some pre-generated anchor frames are removed according to the pixel categories, the second number corresponding to the filtered anchor frames is less than the first number corresponding to the pre-generated anchor frames.

[0032] The above-described method of removing the anchor frame whose pixel category belongs to the background at the center position to obtain the filtered anchor frame is only an example. In another example, according to the actual training or target detection requirements, the anchor frame whose pixel category belongs to the training object or detection object of the user's interest can be retained, and the anchor frames belonging to other categories can be removed to obtain the filtered anchor frame, thereby obtaining a more targeted anchor frame filtering result.

[0033] In addition, in another example, the anchor frame may be filtered according to the comparison between the ratio of pixels with a specific pixel category in the anchor frame and a predetermined threshold value to obtain the filtered anchor frame. Specifically, for example, the ratio of pixels with a background pixel category in the anchor frame relative to all pixels in the anchor frame may be calculated. At this time, if the ratio of pixels with a background pixel category is higher than a predetermined threshold value, the anchor frame may be considered to be an invalid anchor frame in the training process or the target detection process and the anchor frame may be removed accordingly.

[0034] By implementing the above Figure 1 The described image processing method can filter the pre-generated anchor frames according to semantic information to obtain the filtered anchor frames, thereby greatly reducing the number of invalid anchor frames therein, and can use the filtered anchor frames to train the parameters in the neural network during the training process, or send the filtered anchor frames to the trained neural network during the target detection process to obtain the predicted bounding box of the object to be detected, thereby improving the efficiency of training the neural network or performing target detection.

[0035] The above describes in detail the specific process of generating semantic information of the image to be processed and filtering the anchor box using the semantic information. In addition, according to an embodiment of the present invention, the filtered anchor box and the true bounding box can also be used to train a neural network to adjust the parameters for anchor box regression and the parameters for anchor box generation. The following will further describe in detail the specific process of how to train a neural network based on the filtered anchor box to adjust the parameters for anchor box regression and the parameters for anchor box generation.

[0036] Specifically, in one embodiment of the present invention, the training of the neural network using the filtered anchor box and the true bounding box may include: classifying the training object into a first object and a second object according to the ratio of the intersection area to the union area of ​​the filtered anchor box and the true bounding box, training the neural network using the first object to adjust the parameters for anchor box regression, and training the neural network using the second object to adjust the parameters for anchor box generation. The first object constitutes a regular sample for training the neural network to adjust the parameters for anchor box regression, and the second object constitutes a difficult sample for training the neural network to adjust the parameters for anchor box generation.

[0037] Figure 3 FIG. 4 is a flowchart of classifying a training object using a filtered anchor box and a real bounding box of the training object according to an embodiment of the present invention. Figure 3 As shown, at step S301, the ratio of the intersection area to the union area (intersection over union, IOU) of the filtered anchor box and the real bounding box of the training object can be obtained, and at step S302, the obtained IOU can be compared with a predetermined first IOU threshold. When the IOU of the real bounding box of the training object and at least one filtered anchor box is greater than the first IOU threshold, then at step S303, the training object can be classified as a first object for constituting a regular sample; otherwise, if the IOU of the real bounding box of the training object and all the filtered anchor boxes is less than the threshold, then at step S304, the training object can be classified as a second object for constituting a difficult sample.

[0038] As is well known to those skilled in the art, the IOU of the anchor box and the true bounding box can reflect the degree of overlap between the two. The larger the IOU value, the more thorough the overlap between the anchor box and the true bounding box, where IOU=0 means that there is no overlap between the anchor box and the true bounding box, and IOU=1 means that the anchor box and the true bounding box completely overlap. In one example, the first IOU threshold can be set to 0.7. Optionally, the first IOU threshold can also be set to 0.5 or any other reasonable value, and the present invention is not limited to this. Typically, each anchor box is determined to match only a single true bounding box whose IOU meets the threshold requirement and has the maximum IOU, and training is performed by regressing the anchor box to the matched single true bounding box. However, this may result in the existence of other true bounding boxes that, although their IOU meets the threshold requirement with the anchor box, do not have the maximum IOU and therefore cannot participate in the training process. Therefore, in an embodiment of the present invention, in order to maximize the utilization of training samples, when the IOU between the real bounding box of the training object and at least one filtered anchor box is greater than a threshold, it can be determined that the real bounding box matches the at least one filtered anchor box, and the training object can be divided into a first object for constituting a regular sample, so as to be used to train the neural network to adjust the parameters for anchor box regression. However, since the position, size and aspect ratio of the filtered anchor box are predetermined, and the specific size and position of the training object are extremely variable, it may be that the real bounding box of some training objects is greatly different from all the filtered anchor boxes and the IOU cannot meet the threshold requirement, so that the appropriate anchor box cannot be matched. Figure 4 An example of a training object that cannot be matched with a filtered anchor box according to an embodiment of the present invention is shown, wherein a portion of the filtered anchor boxes having a common center position are as follows: Figure 4 As shown in the dashed box in Figure 4 The left side of shows an example where the training object cannot be matched due to the large difference between the shape of the training object and the shape of the filtered anchor box, while Figure 4 The right side of shows another example in which the center of the training object cannot be matched with the center of the filtered anchor box due to a large offset. Accordingly, by dividing the training objects that cannot be matched with the filtered anchor box into second objects for constituting difficult samples and using them to train the neural network to adjust the parameters for generating anchor boxes, the robustness of the neural network against object changes can be enhanced.

[0039] After the training objects are divided into the first objects for constituting regular samples and the second objects for constituting difficult samples in the above manner, the first objects can be used to train the neural network to adjust the parameters for anchor box regression at step S305, or the second objects can be used to train the neural network to adjust the parameters for anchor box generation at step S306. However, it should be noted that the above method of using the second object to train the neural network to adjust the parameters for anchor box generation is only an example. In actual applications, all training objects can also be used to train the neural network together to adjust the parameters for anchor box generation, and the present invention is not limited to this.

[0040] Next, a specific process of using the first object to train the neural network to adjust the parameters for anchor box regression will be described in detail.

[0041] Wherein, the using the first object to train the neural network to adjust the parameters for anchor box regression may include: obtaining a first loss function according to the matching result of the filtered anchor box and the real bounding box of the first object, and using the first loss function to train the neural network to adjust the parameters for anchor box regression. Specifically, during the training process, when the filtered anchor box matches the real bounding box of the first object, the actual anchor box offset of the filtered anchor box relative to the real bounding box matched therewith can be calculated. In addition, the predicted anchor box offset of the filtered anchor box relative to the predicted bounding box can also be calculated, and the predicted bounding box is generated by applying the anchor box regression parameters to the filtered anchor box. Thus, the first loss function can be obtained based on the difference between the predicted anchor box offset and the actual anchor box offset. In one embodiment, the parameters for anchor box regression can be adjusted during the training process to make the obtained first loss function as small as possible, and the parameters for anchor box regression in the neural network when the first loss function converges are used as the adjusted parameters, so that during the target detection process, the anchor box offset predicted according to the adjusted parameters is output, so that the position and size of the pre-generated anchor box are adjusted using the predicted anchor box offset to obtain the predicted bounding box of the object to be detected.

[0042] In an example of constructing the first loss function, the IOU between the filtered anchor box and the true bounding box of the first object is first obtained. Since the IOU between the filtered anchor box and the true bounding boxes of all training objects has been obtained in the aforementioned process of classifying the training objects, in this step, the IOU obtained in the process of classifying the training objects can also be directly used to further reduce the amount of calculation. After obtaining the IOU between the filtered anchor box and the true bounding box of the first object, it is compared with a predetermined second IOU threshold to obtain the matching result of the filtered anchor box and the true bounding box of the first object. Optionally, the second IOU threshold can be equal to or greater than the first IOU threshold for classifying the training object, and when the second IOU threshold is equal to the first IOU threshold, the matching result of the anchor box and the true bounding box obtained in the process of classifying the training object can be directly used as the matching result of the filtered anchor box and the true bounding box of the first object.

[0043] In one example, a first loss function is obtained according to all bounding boxes in the real bounding box of the first object that match the filtered anchor box. As described above, in the usual training process, each anchor box will only match and regress the real bounding box of the first object with the maximum IOU, and accordingly, the first loss function is constructed according to the loss function between the real bounding box of the first object with the maximum IOU and the anchor box. However, in reality, an anchor box may meet the threshold requirement with the IOU of more than one real bounding box of the first object. In the case of constructing the first loss function only according to the loss function between the real bounding box of the first object with the maximum IOU and the anchor box, it may result in a part of the first object that cannot participate in the training process because it does not have the maximum IOU although the IOU with the anchor box meets the threshold requirement. Therefore, in this example, it can be determined that each anchor box matches all real bounding boxes whose IOU meets the threshold requirement, so as to obtain the first loss function based on the loss function between the filtered anchor box and all the real bounding boxes that match it, and use the first loss function for training to maximize the utilization of training samples. Specifically, it is determined that each anchor box matches all real bounding boxes whose IOUs meet the threshold requirements, and a first loss function L1 is obtained according to all bounding boxes in the real bounding box of the first object that match the filtered anchor box, which can be shown as the following formula (1):

[0044]

[0045] The number of the filtered anchor boxes with matching true bounding boxes is represented as n, and the number of all bounding boxes in the true bounding box of the first object that match each anchor box i is represented as m. i , L ij represents the loss function between each anchor box i and each bounding box j in all bounding boxes matching it, IOU ij represents the IOU between each anchor box i and each bounding box j in all bounding boxes that match it, and the IOU i represents the sum of the IOUs between each anchor box i and all bounding boxes that match it. In the first loss function L1 represented by formula (1), the loss function L between each anchor box i and each bounding box j in all bounding boxes that match it is ij The loss weight is the corresponding normalized IOU value IOU ij / IOU i , and by for each loss function L ij A weighted sum is performed to obtain a first loss function L1. Optionally, the loss function L between each anchor box i and each bounding box j in all bounding boxes matching it is ij It can be a cross entropy loss function or a SmoothL1 loss function, but other types of loss functions can also be used, and the present invention is not limited to this. By using all the bounding boxes in the real bounding box of the first object that match the filtered anchor box as described above, a first loss function is obtained to train the neural network to adjust the parameters for anchor box regression so that the obtained first loss function is as small as possible. In this way, it is possible to avoid the existence of a part of objects that cannot participate in the training process even though the IOU with the anchor box meets the threshold requirement, thereby maximizing the utilization of training samples.

[0046] Next, the specific process of using the second object to train the neural network to adjust the parameters for generating anchor boxes will be described in detail.

[0047] Specifically, the use of the second object to train the neural network to adjust the parameters for anchor box generation may include: determining a first area within the true bounding box of the second object according to the semantic segmentation area, and generating a second loss function according to the pixels within the first area, so as to use the second loss function to train the neural network to adjust the parameters for anchor box generation. Specifically, the second loss function may reflect the difference between the anchor box predicted according to the parameters for anchor box generation and the true bounding box of the training object. In one embodiment, the parameters for anchor box generation in the neural network may be adjusted during the training process to make the second loss function as small as possible, and the parameters for anchor box generation in the neural network when the second loss function converges are used as the adjusted parameters, thereby outputting the anchor box predicted according to the adjusted parameters during the target detection process. In one example, based on the target detection technology without anchor boxes, the anchor box predicted according to the parameters for anchor box generation may be directly used as the predicted bounding box of the final object to be detected. In another example, according to the anchor box based target detection technology, the anchor box offset can be predicted by the parameters used for anchor box regression, so that the position and size of the anchor box predicted according to the parameters used for anchor box generation can be further adjusted to obtain the predicted bounding box of the object to be detected, or the anchor box predicted according to the parameters used for anchor box generation can be first filtered according to semantic information, and then the anchor box offset can be predicted by the parameters used for anchor box regression, so that the position and size of the filtered anchor box can be further adjusted to obtain the predicted bounding box of the object to be detected.

[0048] Figure 5 FIG. 2 is a schematic diagram showing a prediction result output according to parameters for generating anchor boxes according to an embodiment of the present invention. Figure 5 As shown in FIG. 1 , the prediction result outputted according to the parameters used to generate the anchor box can be the distance from one pixel to the top (t), bottom (b), left (l) and right (r) boundaries of the anchor box, and accordingly, when the distance from one pixel to the four boundaries of the anchor box is predicted, it is equivalent to determining the size, dimension and position of the anchor box, thereby obtaining the following Figure 5 However, the present invention is not limited thereto. In another example, the prediction result output according to the parameters used to generate the anchor box may also be an offset of one pixel from the center position of the anchor box and the size of the anchor box, and the present invention is not limited thereto.

[0049] In the above Figure 2In the example of generating semantic information described, it has been explained that the pixel category of each pixel on the image to be processed can also correspond to the semantic segmentation area on the image to be processed. Accordingly, the position of the pixel corresponding to the second object and the position of the pixel not corresponding to the second object in the real bounding box of the second object can be distinguished according to the pixel category and the semantic segmentation area. Thus, the first area corresponding to the position of the pixel of the second object in the real bounding box of the second object can be determined, so as to train the neural network with the pixels in the first area to adjust the parameters for anchor frame generation, thereby effectively reducing the amount of calculation in the training process to improve training efficiency. Specifically, in one example, when the pixel category of each pixel on the image to be processed is roughly predicted as "foreground" or "background", the semantic segmentation area corresponding to the "background" pixel category in the real bounding box of the second object can be removed, and the remaining area in the real bounding box is used as the first area, so as to train the neural network with only the pixels in the first area to adjust the parameters for anchor frame generation. In another example, when each pixel on the image to be processed is divided into a more specific pixel category related to an object category, a semantic segmentation area corresponding to a pixel category consistent with the object category of the second object within the true bounding box of the second object can be determined as the first area, so that only pixels within the first area are used to train the neural network to adjust the parameters for anchor box generation.

[0050] In this example, after determining the first region within the true bounding box of the second object, a second loss function can be generated according to the pixels within the first region, so as to train the neural network using the second loss function to adjust the parameters for anchor box generation. Specifically, the second loss function is composed of a loss function of each pixel within the first region, wherein the loss function of each pixel within the first region can reflect the difference between the anchor box predicted for each pixel within the first region using the parameters for anchor box generation and the true bounding box of the training object.

[0051] By using the second object constituting a difficult sample to train a neural network to adjust the parameters for generating an anchor box as described above, and determining a first area within a true bounding box of the second object based on the semantic segmentation area, and generating a second loss function for training based on pixels within the first area, the robustness of the neural network to object changes can be enhanced, and the amount of computation in the training process can be effectively reduced to improve training efficiency.

[0052] According to the above image processing method of the present invention, a large number of pre-generated anchor frames can be filtered according to semantic information to remove invalid anchor frames, so that the filtered anchor frames are further used to train neural networks and perform target detection. This image processing method can reduce the processing of invalid anchor frames in the training process or target detection process, and improve the efficiency of training neural networks and performing target detection.

[0053] In addition, according to the above-mentioned image processing method of the present invention, the training objects can be divided into regular samples for training the neural network to adjust the parameters for anchor box regression, and difficult samples for training the neural network to adjust the parameters for anchor box generation, thereby enhancing the robustness of the neural network to object changes.

[0054] The following will refer to Figure 6 An image processing apparatus according to an embodiment of the present invention is described. Figure 6 FIG. 6 is a block diagram of an image processing apparatus 600 according to an embodiment of the present invention. Figure 6 As shown, the image processing device 600 includes an acquisition unit 610, a semantic information generation unit 620, an anchor frame filtering unit 630, an object classification unit 640, an anchor frame regression unit 650, and an anchor frame generation unit 660. Here, each component of the image processing device 600 can implement the corresponding function or operation of the neural network according to the embodiment of the present invention. In addition to these units, the image processing device 600 may also include other components, for example, Figure 6 The image processing device shown may also include units for implementing other functions or operations of the neural network, which are not limited by the present invention. Figure 1-Figure 5 The details of the description are the same, so they are briefly described here to avoid repetition.

[0055] The acquisition unit 610 in the image processing device 600 is configured to acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number.

[0056] As is well known to those skilled in the art, the process of using a neural network for target detection can generally be divided into two specific processes: a "training process" and a "target detection process". Accordingly, the image to be processed acquired by the acquisition unit 610 may be a training image for training a neural network, or a detection image for target detection. When the image to be processed is a training image, at least one object contained in the image to be processed is a training object, and the training object is pre-annotated with a training object category and a real bounding box surrounding the training object to indicate its specific position on the image to be processed; when the image to be processed is a detection image, the image to be processed may contain at least one object to be detected, and the target detection result may be a predicted category of the object to be detected output on the image to be processed and a predicted bounding box surrounding the object to be detected to indicate its specific position on the image to be processed.

[0057] The acquisition unit 610 can acquire multiple pre-generated anchor frames on the image to be processed as a training image or a detection image, and use these pre-generated anchor frames together with the real bounding box of the training object to train the neural network in the case of training, or, in the case of target detection, send these pre-generated anchor frames to the trained neural network to obtain the final predicted bounding box of the object to be detected. Optionally, during the training process or the target detection process, rectangular frames with different positions, sizes and aspect ratios are generated as the multiple pre-generated anchor frames with each pixel or part of the pixel on the image to be processed as the center. In one example, the position, size and aspect ratio of the pre-generated anchor frame can be manually set. In another example, an anchor frame that is more in line with the training object or the object to be detected can be automatically generated by a k-means clustering algorithm or the like.

[0058] For ease of explanation, the number of anchor frames pre-generated in this step is referred to as the first number. Generally, in order to better cover and match real bounding boxes with various sizes and positions or obtain predicted bounding boxes of objects to be detected with different positions, sizes and shapes, the pre-generated anchor frames are often very dense and large in number. In contrast, the training objects contained in the training images or the objects to be detected contained in the detection images are usually sparse, so the number of anchor frames that can be matched with the training objects and actually participate in the training during the training process, or the number of anchor frames through which the final predicted bounding boxes can be generated during the target detection process is actually very limited, which results in a large amount of time and calculation being wasted in the training process or the target detection process to process invalid anchor frames, thereby affecting the efficiency of training the neural network or performing target detection. In an embodiment of the present invention, the number of invalid anchor frames can be reduced by filtering the pre-generated anchor frames according to the semantic information described in detail below, thereby improving the efficiency of training the neural network or performing target detection.

[0059] The semantic information generating unit 620 in the image processing device 600 is configured to generate semantic information according to the image to be processed.

[0060] Specifically, the semantic information generating unit 620 generates semantic information according to the image to be processed, so that in a subsequent process, the anchor frame filtering unit 630 filters the pre-generated anchor frame according to the semantic information. In an example, the semantic information generating unit 620 generates semantic information according to the image to be processed, which may include: obtaining a pixel category of each pixel on the image to be processed according to the image to be processed.

[0061] For example, Figure 2 As shown, the semantic information generating unit 620 can generate Figure 2 The semantic information of the image to be processed on the left side is predicted, and the prediction result shown in the image on the right is obtained, wherein each pixel on the image to be processed is predicted to have the pixel category "person", "car" or "background". The above-mentioned method of predicting each pixel on the image to be processed as having a pixel category such as "person", "car" or "background" by the semantic information generation unit 620 is only an example. In actual applications, the semantic information generation unit 620 can only roughly predict the pixel category of each pixel on the image to be processed as "foreground" or "background" so as not to generate pixel categories involving specific object categories, or can make a more detailed prediction and division of the pixel category of each pixel on the image to be processed and generate a more specific pixel category, so that the pixel category is adapted to the category of the training object or detection object that the user is concerned about, so as to meet different anchor box filtering requirements.

[0062] The anchor box filtering unit 630 in the image processing device 600 is configured to filter the anchor boxes using the semantic information to obtain filtered anchor boxes, wherein the number of the filtered anchor boxes is a second number, which is smaller than the first number.

[0063] When the predicted semantic information is outputted by the semantic information generation unit 620 to obtain the pixel category of each pixel on the image to be processed, the anchor frame filtering unit 630 can filter the pre-generated anchor frame based on the pixel category, and remove part of the anchor frame according to the pixel category to obtain the filtered anchor frame.

[0064] In one example, the anchor frame filtering unit 630 can filter the anchor frames according to the pixel categories of the pixels at the center position of each anchor frame. For example, when the pixel category of the pixel at the center position of the anchor frame belongs to "background", it can be considered that the offset of the anchor frame from the real bounding box of the training object pre-annotated in the training process or the predicted bounding box of the object to be detected finally generated in the target detection process is large, and accordingly belongs to an invalid anchor frame in the training process or the target detection process. Therefore, in this example, the anchor frames can be filtered according to the pixel categories of the pixels at the center position of each anchor frame, so as to remove the anchor frames whose pixel categories of the pixels at the center position belong to the background to obtain the filtered anchor frames. The number of filtered anchor frames obtained in this step is called the second number, and since some pre-generated anchor frames are removed according to the pixel categories, the second number corresponding to the filtered anchor frames is less than the first number corresponding to the pre-generated anchor frames.

[0065] The above-described method of removing the anchor frame whose pixel category belongs to the background at the center position to obtain the filtered anchor frame is only an example. In another example, the anchor frame filtering unit 630 can retain the anchor frame whose pixel category belongs to the training object or detection object concerned by the user according to the actual training or target detection requirements, and remove the anchor frames belonging to other categories to obtain the filtered anchor frame, so as to obtain a more targeted anchor frame filtering result.

[0066] In addition, in another example, the anchor frame filtering unit 630 can also filter the anchor frame according to the comparison between the proportion of pixels with a specific pixel category in the anchor frame and a predetermined threshold value to obtain the filtered anchor frame. Specifically, for example, the proportion of pixels with a background pixel category in the anchor frame relative to all pixels in the anchor frame can be calculated. At this time, if the proportion of pixels with a background pixel category is higher than a predetermined threshold value, the anchor frame can be considered to be an invalid anchor frame in the training process or the target detection process and the anchor frame can be removed accordingly.

[0067] According to the above-mentioned image processing device of the present invention, the pre-generated anchor frames can be filtered according to semantic information to obtain the filtered anchor frames, thereby greatly reducing the number of invalid anchor frames therein, and can use the filtered anchor frames to train the parameters in the neural network during the training process, or send the filtered anchor frames to the trained neural network during the target detection process to obtain the predicted bounding box of the object to be detected, thereby improving the efficiency of training the neural network or performing target detection.

[0068] The above describes in detail the specific process of using the semantic information generation unit 620 included in the above-mentioned image processing device of the present invention to generate the semantic information of the image to be processed, so that the anchor frame filtering unit 630 uses the semantic information to filter the anchor frame. In addition, during the training process, the object classification unit 640 can also be used to classify the training objects, and the classified training objects can be used to train the neural network through the anchor frame regression unit 650 and the anchor frame generation unit 660 to respectively adjust the parameters for anchor frame regression and the parameters for anchor frame generation. It should be noted here that the above method of training the neural network using the object classification unit 640, the anchor frame regression unit 650 and the anchor frame generation unit 660 is only an example. In actual applications, the anchor frame regression unit 650 and the anchor frame generation unit 660 can also be applied to the actual target detection process, and the present invention is not limited to this.

[0069] Specifically, in one embodiment of the present invention, the object classification unit 640 may classify the training object into a first object and a second object according to the ratio of the intersection area to the union area of ​​the filtered anchor box and the true bounding box, the anchor box regression unit 650 may use the first object to train the neural network to adjust the parameters for anchor box regression, and the anchor box generation unit 660 may use the second object to train the neural network to adjust the parameters for anchor box generation. The first object constitutes a regular sample for training the neural network to adjust the parameters for anchor box regression, and the second object constitutes a difficult sample for training the neural network to adjust the parameters for anchor box generation.

[0070] like Figure 3 As shown, the object classification unit 640 can obtain the ratio of the intersection area to the union area (IOU) of the filtered anchor box and the real bounding box of the training object, and compare the obtained IOU with a predetermined first IOU threshold. When the IOU of the real bounding box of the training object and at least one filtered anchor box is greater than the first IOU threshold, the object classification unit 640 can classify the training object as a first object for constituting a regular sample; otherwise, if the IOU of the real bounding box of the training object and all the filtered anchor boxes is less than the threshold, the object classification unit 640 can classify the training object as a second object for constituting a difficult sample.

[0071] Typically, during the training process, each anchor box is determined to match only a single true bounding box whose IOU meets the threshold requirement and has the maximum IOU, and the training is performed by regressing the anchor box to the matched single true bounding box. However, this may result in the existence of other true bounding boxes that meet the threshold requirement with the anchor box but do not have the maximum IOU and therefore cannot participate in the training process. Therefore, in an embodiment of the present invention, in order to maximize the utilization of training samples, the object classification unit 640 can determine that the true bounding box matches the at least one filtered anchor box when the IOU of the true bounding box of the training object is greater than the threshold with the at least one filtered anchor box, and can divide the training object into a first object for constituting a regular sample for training by the anchor box regression unit 650. In addition, the object classification unit 640 can enhance the robustness of the neural network to object changes by dividing the training objects that cannot be matched with the filtered anchor box into a second object for constituting a difficult sample and for training by the anchor box generation unit 660.

[0072] After the object classification unit 640 divides the training objects into the first objects for constituting regular samples and the second objects for constituting difficult samples in the above manner, the anchor frame regression unit 650 can correspondingly use the first objects to train the neural network to adjust the parameters for anchor frame regression, and the anchor frame generation unit 660 can correspondingly use the second objects to train the neural network to adjust the parameters for anchor frame generation. However, it should be noted that the above method in which the anchor frame generation unit 660 uses the second object to train the neural network to adjust the parameters for anchor frame generation is only an example. In actual applications, the anchor frame generation unit 660 can also use all training objects to train the neural network together to adjust the parameters for anchor frame generation, and the present invention is not limited to this.

[0073] Next, the specific process of the anchor box regression unit 650 using the first object to train the neural network to adjust the parameters for anchor box regression will be described in detail.

[0074] Among them, the anchor frame regression unit 650 uses the first object to train the neural network to adjust the parameters for anchor frame regression, which may include: obtaining a first loss function according to the matching result between the filtered anchor frame and the real bounding box of the first object, and using the first loss function to train the neural network to adjust the parameters for anchor frame regression. Specifically, during the training process, when the filtered anchor frame matches the real bounding box of the first object, the anchor frame regression unit 650 can calculate the actual anchor frame offset of the filtered anchor frame relative to the real bounding box matched therewith. In addition, the anchor frame regression unit 650 can also calculate the predicted anchor frame offset of the filtered anchor frame relative to the predicted bounding box, and the predicted bounding box is generated by applying the anchor frame regression parameters to the filtered anchor frame. Thus, the first loss function can be obtained based on the difference between the predicted anchor frame offset and the actual anchor frame offset. In one embodiment, the anchor frame regression unit 650 can adjust the parameters used for anchor frame regression during the training process to make the obtained first loss function as small as possible, and use the parameters used for anchor frame regression in the neural network when the first loss function converges as the adjusted parameters, thereby outputting the anchor frame offset predicted according to the adjusted parameters during the target detection process, and using the predicted anchor frame offset to adjust the position and size of the pre-generated anchor frame to obtain the predicted bounding box of the object to be detected.

[0075] In an example of constructing the first loss function, the anchor box regression unit 650 first obtains the IOU between the filtered anchor box and the real bounding box of the first object. Since the IOU between the filtered anchor box and the real bounding box of all training objects has been obtained in the aforementioned process of classifying the training objects, in this step, the IOU obtained in the process of classifying the training objects can also be directly used to further reduce the amount of calculation. After obtaining the IOU between the filtered anchor box and the real bounding box of the first object, it is compared with a predetermined second IOU threshold to obtain the matching result of the filtered anchor box and the real bounding box of the first object. Optionally, the second IOU threshold can be equal to or greater than the first IOU threshold for classifying the training object, and when the second IOU threshold is equal to the first IOU threshold, the matching result of the anchor box and the real bounding box obtained in the process of classifying the training object can be directly used as the matching result of the filtered anchor box and the real bounding box of the first object.

[0076] In one example, the anchor frame regression unit 650 obtains a first loss function according to all bounding boxes in the real bounding box of the first object that match the filtered anchor frame. As described above, in the usual training process, each anchor frame will only match and regress the real bounding box of the first object with the maximum IOU, and accordingly, the first loss function is constructed according to the loss function between the real bounding box of the first object with the maximum IOU and the anchor frame. However, in reality, an anchor frame may meet the threshold requirement for the IOU with more than one real bounding box of the first object. In the case where the first loss function is constructed only according to the loss function between the real bounding box of the first object with the maximum IOU and the anchor frame, it may result in a part of the first object that cannot participate in the training process because it does not have the maximum IOU although the IOU with the anchor frame meets the threshold requirement. Therefore, in this example, it can be determined that each anchor frame matches all real bounding boxes whose IOU meets the threshold requirement, so as to obtain the first loss function based on the loss function between the filtered anchor frame and all the real bounding boxes that match it, and use the first loss function for training to maximize the utilization of training samples. Specifically, it is determined that each anchor box matches all real bounding boxes whose IOUs meet the threshold requirements, and a first loss function L1 is obtained according to all bounding boxes in the real bounding box of the first object that match the filtered anchor box, which can be shown as the following formula (1):

[0077]

[0078] The number of the filtered anchor boxes with matching true bounding boxes is represented as n, and the number of all bounding boxes in the true bounding box of the first object that match each anchor box i is represented as m. i , L ij represents the loss function between each anchor box i and each bounding box j in all bounding boxes matching it, IOU ij represents the IOU between each anchor box i and each bounding box j in all bounding boxes that match it, and the IOU i represents the sum of the IOUs between each anchor box i and all bounding boxes that match it. In the first loss function L1 represented by formula (1), the loss function L between each anchor box i and each bounding box j in all bounding boxes that match it is ij The loss weight is the corresponding normalized IOU value IOU ij / IOU i , and by for each loss function L ij A weighted sum is performed to obtain a first loss function L1. Optionally, the loss function L between each anchor box i and each bounding box j in all bounding boxes matching it isij It can be a cross entropy loss function or a SmoothL1 loss function, but other types of loss functions can also be used, and the present invention is not limited to this. The anchor box regression unit 650 described above uses all bounding boxes in the real bounding box of the first object that match the filtered anchor box to obtain a first loss function to train the neural network to adjust the parameters for anchor box regression so that the obtained first loss function is as small as possible. In this way, it is possible to avoid the existence of a part of objects that cannot participate in the training process even though their IOU with the anchor box meets the threshold requirement, thereby maximizing the utilization of training samples.

[0079] Next, the specific process in which the anchor box generation unit 660 uses the second object to train the neural network to adjust the parameters for anchor box generation will be described in detail.

[0080] Specifically, the anchor frame generation unit 660 uses the second object to train the neural network to adjust the parameters for anchor frame generation, which may include: determining a first area within a true bounding box of the second object according to the semantic segmentation area, and generating a second loss function according to pixels within the first area, so as to train the neural network using the second loss function to adjust the parameters for anchor frame generation.

[0081] In the above Figure 2In the example of generating semantic information described, it has been explained that the pixel category of each pixel on the image to be processed can also correspond to the semantic segmentation area on the image to be processed. Accordingly, the anchor frame generation unit 660 can distinguish the position of the pixel corresponding to the second object and the position of the pixel not corresponding to the second object in the real bounding box of the second object according to the pixel category and the semantic segmentation area. Thus, the anchor frame generation unit 660 can determine the first area of ​​the position of the pixel corresponding to the second object in the real bounding box of the second object, so as to train the neural network with the pixels in the first area, thereby effectively reducing the amount of calculation in the training process to improve the training efficiency. Specifically, in one example, when the pixel category of each pixel on the image to be processed is roughly predicted as "foreground" or "background", the anchor frame generation unit 660 can remove the semantic segmentation area corresponding to the "background" pixel category in the real bounding box of the second object, and use the remaining area in the real bounding box as the first area, so as to train the neural network only with the pixels in the first area to adjust the parameters for anchor frame generation. In another example, when each pixel on the image to be processed is divided into a more specific pixel category related to an object category, the anchor frame generation unit 660 can determine the semantic segmentation area corresponding to the pixel category consistent with the object category of the second object within the true bounding box of the second object as the first area, thereby training the neural network using only the pixels within the first area to adjust the parameters for anchor frame generation.

[0082] In this example, after determining the first area within the true bounding box of the second object, the anchor box generation unit 660 can generate a second loss function based on the pixels within the first area, so as to train the neural network using the second loss function to adjust the parameters for anchor box generation. Specifically, the second loss function is composed of a loss function for each pixel in the first area, wherein the loss function for each pixel in the first area can reflect the difference between the anchor box predicted for each pixel in the first area using the parameters for anchor box generation and the true bounding box of the training object.

[0083] By using the anchor frame generation unit 660 described above to train the neural network using the second object constituting the difficult sample, and determining the first area within the true bounding box of the second object based on the semantic segmentation area, and generating a second loss function for training based on the pixels within the first area, the robustness of the neural network against object changes can be enhanced, and the amount of computation in the training process can be effectively reduced to improve training efficiency.

[0084] In one example, when the semantic information generation unit 620, the anchor box regression unit 650, and the anchor box generation unit 660 are applied to the same neural network, the three units can share shallow layers in the neural network such as those used for feature extraction by sharing the same parameters, thereby improving the training efficiency of the three units.

[0085] According to the above-mentioned image processing device of the present invention, a large number of pre-generated anchor frames can be filtered according to semantic information to remove invalid anchor frames therein, so that the filtered anchor frames are further used for training neural networks and performing target detection. This image processing device can reduce the processing of invalid anchor frames in the training process or target detection process, and improve the efficiency of training neural networks and performing target detection.

[0086] In addition, according to the above-mentioned image processing device of the present invention, the training objects can be divided into regular samples for training the neural network to adjust the parameters for anchor box regression, and difficult samples for training the neural network to adjust the parameters for anchor box generation, thereby enhancing the robustness of the neural network against object changes.

[0087] Below, refer to Figure 7 An image processing apparatus according to an embodiment of the present invention will be described. Figure 7 FIG. 7 is a block diagram of an image processing apparatus 700 according to an embodiment of the present invention. Figure 7 As shown, the device 700 may be a computer or a server.

[0088] like Figure 7 As shown, the image processing device 700 includes one or more processors 710 and a memory 720. Of course, in addition to this, the image processing device 700 may also include a multi-sensor imaging system and an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Figure 7 The components and structures of the image processing device 700 shown are merely exemplary and not restrictive. The image processing device 700 may also have other components and structures as required.

[0089] The processor 710 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 720 to perform desired functions, which may include: acquiring an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information based on the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is less than the first number.

[0090] The memory 720 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 710 may run the program instructions to implement the functions of the image processing device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the image processing method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0091] Below, a computer-readable storage medium according to an embodiment of the present invention is described, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; generating semantic information according to the image to be processed; and filtering the anchor frames using the semantic information to obtain filtered anchor frames, wherein the number of the filtered anchor frames is a second number, and the second number is less than the first number.

[0092] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0093] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0094] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0095] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to an element in the singular, such as using "one", "one", or "the" is not to be construed as limiting the element to the singular.

[0096] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0097] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0098] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0099] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0100] The method invented herein includes one or more actions for implementing the method described. Methods and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0101] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0102] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0103] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0104] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0105] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0106] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0107] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0108] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image processing method, comprising: Acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; Acquire a pixel category of each pixel on the image to be processed according to the image to be processed; as well as filtering the anchor frames according to the pixel categories of at least part of the pixels on the image to be processed to obtain a second number of filtered anchor frames, wherein the second number is smaller than the first number; The image to be processed is a training image, the at least one object is a training object, the training image is pre-annotated with a real bounding box of each training object, and the method further comprises training a neural network using the filtered anchor box and the real bounding box; The step of training a neural network using the filtered anchor box and the real bounding box comprises: According to the ratio of the intersection area to the union area of ​​the filtered anchor box and the true bounding box, the training object is classified into a first object and a second object, the first object is used to train the neural network to adjust the parameters for anchor box regression, and the second object is used to train the neural network to adjust the parameters for anchor box generation.

2. The method of claim 1, wherein: The using the first object to train the neural network to adjust the parameters for anchor box regression includes: obtaining a first loss function according to a matching result between the filtered anchor box and a real bounding box of the first object, and using the first loss function to train the neural network to adjust the parameters for anchor box regression.

3. The method of claim 2, wherein: A first loss function is obtained based on all bounding boxes in the real bounding box of the first object that match the filtered anchor box.

4. The method of claim 3, wherein: The pixel category of each pixel on the image to be processed corresponds to the semantic segmentation region on the image to be processed, The using the second object to train the neural network to adjust the parameters for anchor box generation includes: determining a first area within a true bounding box of the second object according to the semantic segmentation area, and generating a second loss function according to pixels within the first area, so as to use the second loss function to train the neural network to adjust the parameters for anchor box generation.

5. The method according to any one of claims 1 to 4, wherein: The method also includes obtaining a predicted bounding box of at least one object to be detected on the detection image according to the neural network.

6. An image processing device, comprising: an acquisition unit, configured to acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; A semantic information generating unit, configured to obtain a pixel category of each pixel on the image to be processed according to the image to be processed; as well as an anchor frame filtering unit, configured to filter the anchor frames according to the pixel categories of at least part of the pixels on the image to be processed, so as to obtain a second number of filtered anchor frames, wherein the second number is smaller than the first number; The image to be processed is a training image, the at least one object is a training object, the training image is pre-annotated with a real bounding box of each training object, and the image processing device is further configured to train a neural network using the filtered anchor box and the real bounding box; The image processing device further comprises: an object classification unit configured to classify the training object into a first object and a second object according to a ratio of an intersection area to a union area of ​​the filtered anchor box and the true bounding box; an anchor box regression unit configured to train the neural network using the first object to adjust parameters for anchor box regression; and An anchor box generating unit is configured to train the neural network using the second object to adjust parameters for generating an anchor box.

7. An image processing device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; Acquire a pixel category of each pixel on the image to be processed according to the image to be processed; and filtering the anchor frames according to the pixel categories of at least part of the pixels on the image to be processed to obtain a second number of filtered anchor frames, wherein the second number is smaller than the first number; The image to be processed is a training image, the at least one object is a training object, and the training image is pre-marked with a real boundary box of each training object. When the computer program instructions are executed by the processor, the processor further performs the step of training a neural network using the filtered anchor box and the true bounding box, including: According to the ratio of the intersection area to the union area of ​​the filtered anchor box and the true bounding box, the training object is classified into a first object and a second object, the first object is used to train the neural network to adjust the parameters for anchor box regression, and the second object is used to train the neural network to adjust the parameters for anchor box generation.

8. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Acquire an image to be processed and a plurality of pre-generated anchor frames on the image to be processed, wherein the image to be processed is an image containing at least one object, and the number of the anchor frames is a first number; Acquire a pixel category of each pixel on the image to be processed according to the image to be processed; as well as filtering the anchor frames according to the pixel categories of at least part of the pixels on the image to be processed to obtain a second number of filtered anchor frames, wherein the second number is smaller than the first number; The image to be processed is a training image, the at least one object is a training object, and the training image is pre-marked with a real boundary box of each training object. When the computer program instructions are executed by the processor, the steps of training a neural network using the filtered anchor box and the true bounding box are implemented, including: According to the ratio of the intersection area to the union area of ​​the filtered anchor box and the true bounding box, the training object is classified into a first object and a second object, the first object is used to train the neural network to adjust the parameters for anchor box regression, and the second object is used to train the neural network to adjust the parameters for anchor box generation.

Images