Image processing method based on ATSS, device, electronic device, and storage medium
By using the Adaptive Training Sample Selection Algorithm (ATSS) in the Transformer model to filter candidate boxes and generate tokens, the problem of high computational cost of the Transformer model is solved, and the efficiency of image processing is improved.
Patent Information
- Application Number
- CN202111277386.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-10-29
AI Technical Summary
Transformer models are computationally intensive in image recognition, which affects efficiency, especially since the self-attention mechanism requires calculating the self-attention weights of the entire feature map.
An adaptive training sample selection algorithm (ATSS) is adopted. Candidate boxes are generated in the feature map, positive samples that meet the conditions are selected, tokens are generated for self-attention weight calculation, and candidate boxes that do not meet the conditions are discarded to reduce invalid calculations.
This effectively reduces the computational cost of tokens, improves the efficiency of image processing, and enhances the performance of the Transformer model.
Smart Images

Figure CN113989493B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to but is not limited to the field of artificial intelligence, and in particular to an ATSS-based image processing method and its device, electronic device, and storage medium. Background Art
[0002] With the development of artificial intelligence technology, more and more models are applied to image recognition, and the Transformer model has powerful performance and is increasingly widely used. Although the Transformer model has a very high accuracy, its core is the self-attention mechanism. After the image is input, the self-attention weight needs to be calculated using the entire feature map as a token, and then fused with the input image based on the self-attention weight. The time complexity of the Transformer model is very high, which will result in a large amount of calculation and affect the efficiency of image recognition. Summary of the invention
[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
[0004] The embodiments of the present invention provide an ATSS-based image processing method and device, an electronic device, and a storage medium, which can reduce the amount of token calculation and thus improve the efficiency of the Transformer model.
[0005] In a first aspect, an embodiment of the present invention provides an image processing method based on ATSS, comprising:
[0006] Acquire a feature map, wherein the feature map is pre-marked with an original marking frame;
[0007] Obtaining a downsampling scale of the feature map, and generating a plurality of candidate boxes in the feature map according to the downsampling scale;
[0008] Determine a first distance between the candidate frame and the original marked frame, and when the first distance satisfies a preset first condition, determine that the candidate frame is a valid candidate frame;
[0009] Determine an intersection-and-union ratio between the valid candidate frame and the original marked frame, and when the intersection-and-union ratio satisfies a preset second condition, determine that the valid candidate frame is a positive sample;
[0010] A token is generated according to the positive sample, and a self-attention weight is calculated according to the token.
[0011] In some embodiments, determining a first distance between the candidate box and the original marked box includes:
[0012] Obtaining a first coordinate and a second coordinate, wherein the first coordinate is the coordinate of the center point of the candidate frame, and the second coordinate is the coordinate of the center point of the original marked frame;
[0013] A distance between the first coordinate and the second coordinate is determined as the first distance.
[0014] In some embodiments, determining the intersection-over-union ratio between the valid candidate box and the original marked box includes:
[0015] Determine a first area, where the first area is an intersection area of the valid candidate box and the original marked box;
[0016] Determine a second area, where the second area is a union area of the valid candidate box and the original marked box;
[0017] The ratio of the area of the second region to the area of the first region is determined as the intersection-over-union ratio.
[0018] In some embodiments, obtaining the downsampling scale of the feature map includes:
[0019] Determine the current sampling depth of the feature map as a target sampling depth;
[0020] Acquire a preset scale mapping table, wherein the scale mapping table records a mapping relationship between a sampling depth and the downsampling scale;
[0021] The downsampling scale is determined from the scale mapping table according to the target sampling depth.
[0022] In some embodiments, generating a plurality of candidate boxes in the feature map according to the downsampling scale includes:
[0023] The sampling width and sampling length are obtained according to the downsampling scale
[0024] At least two candidate boxes are cut out from the feature map according to the sampling width and the sampling length.
[0025] In some embodiments, the scale mapping table includes at least a first downsampling scale and a second downsampling scale, wherein the second downsampling scale is larger than the first downsampling scale, and the sampling depth corresponding to the second downsampling scale is larger than the sampling depth corresponding to the first downsampling scale.
[0026] In some embodiments, after generating a plurality of candidate boxes in the feature map according to the downsampling scale, the method further includes:
[0027] When there is no candidate box whose first distance satisfies the first condition, and / or there is no candidate box whose intersection-over-union ratio satisfies the second condition, the sampling depth of the feature map is increased.
[0028] In a second aspect, an embodiment of the present invention further provides an ATSS-based image processing device, comprising:
[0029] An acquisition unit, used for acquiring a feature map, wherein the feature map is pre-marked with an original marking frame;
[0030] A sampling unit, used to obtain a downsampling scale of the feature map, and generate a plurality of candidate boxes in the feature map according to the downsampling scale;
[0031] A first determining unit, configured to determine a first distance between the candidate frame and the original marked frame, and determine that the candidate frame is a valid candidate frame when the first distance satisfies a preset first condition;
[0032] A second determination unit is used to determine an intersection-over-union ratio between the valid candidate frame and the original marked frame, and when the intersection-over-union ratio satisfies a preset second condition, determine that the valid candidate frame is a positive sample;
[0033] A calculation unit is used to generate a token according to the positive sample and calculate a self-attention weight according to the token.
[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the ATSS-based image processing method as described in the first aspect is implemented.
[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the ATSS-based image processing method as described in the first aspect.
[0036] The embodiment of the present invention includes: obtaining a feature map, wherein the feature map is pre-marked with an original label frame; obtaining a downsampling scale of the feature map, and generating multiple candidate frames in the feature map according to the downsampling scale; determining a first distance between the candidate frame and the original label frame, and determining that the candidate frame is a valid candidate frame when the first distance satisfies a pre-set first condition; determining an intersection-and-union ratio between the valid candidate frame and the original label frame, and determining that the valid candidate frame is a positive sample when the intersection-and-union ratio satisfies a pre-set second condition; generating a token according to the positive sample, and calculating a self-attention weight according to the token. According to the technical solution of this embodiment, the candidate frames can be screened by the set first and second conditions, and the candidate frames that do not meet the conditions can be discarded. The token calculation amount is effectively reduced by token cropping, and the efficiency of image processing is improved.
[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.
[0039] Figure 1 is a flow chart of an image processing method based on ATSS provided by an embodiment of the present invention;
[0040] Figure 2 is a flow chart of determining a first distance provided by another embodiment of the present invention;
[0041] Figure 3 is a flow chart for determining an intersection-and-union ratio provided by another embodiment of the present invention;
[0042] Figure 4 is a flowchart of determining a downsampling scale provided by another embodiment of the present invention;
[0043] Figure 5 is a flow chart of a cutting feature graph provided by another embodiment of the present invention;
[0044] Figure 6 is an example diagram of adjusting the depth provided by another embodiment of the present invention;
[0045] Figure 7 is an example diagram of a feature diagram provided by another embodiment of the present invention;
[0046] Figure 8 is a structural diagram of an ATSS-based image processing device provided by another embodiment of the present invention;
[0047] Fig. 9 is a device diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0049] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0050] The present invention provides an image processing method based on ATSS and its device, electronic device, and storage medium. The image processing method based on ATSS includes: obtaining a feature map, wherein the feature map is pre-marked with an original labeling box; obtaining a downsampling scale of the feature map, and generating multiple candidate boxes in the feature map according to the downsampling scale; determining a first distance between the candidate box and the original labeling box, and when the first distance meets a first condition set in advance, determining that the candidate box is a valid candidate box; determining the intersection-and-union ratio between the valid candidate box and the original labeling box, and when the intersection-and-union ratio meets a second condition set in advance, determining that the valid candidate box is a positive sample; generating a token according to the positive sample, and calculating the self-attention weight according to the token. According to the technical solution of this embodiment, the candidate boxes can be screened by the set first and second conditions, and the candidate boxes that do not meet the conditions are discarded. The calculation amount of the token is effectively reduced by token clipping, and the efficiency of image processing is improved.
[0051] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0052] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0053] Computer vision technology (Computer Vision, CV) is a science that studies how to make machines "see". To put it more concretely, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The image classification method involved in the embodiment of the present application, that is, the application of computer vision technology in the field of image recognition, can reduce the demand for labeled sample images during the training of image classification models, and help improve the accuracy of the prediction results of image classification models after training.
[0054] The self-attention mechanism simulates the human information processing process: due to the limitations of information processing capabilities, humans will selectively focus on some information when processing information, while ignoring or weakening other parts of information. By introducing the attention mechanism, the performance of the neural network can be greatly improved.
[0055] The execution subject of the above-mentioned image processing method may be a computer device, which may be a terminal or a server. The terminal mentioned here may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a car computer, a smart home, a wearable electronic device, a VR (Virtual Reality) / AR (Augmented Reality) device, etc.; the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms and other basic cloud computing services, etc.
[0056] It should be noted that the data of the embodiments of the present invention can be stored in a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0057] like Figure 1 As shown, Figure 1 An ATSS-based image processing method provided by an embodiment of the present invention includes but is not limited to the following steps:
[0058] Step S110, obtaining a feature map, wherein the feature map is pre-marked with an original marking box;
[0059] Step S120, obtaining a downsampling scale of the feature map, and generating multiple candidate boxes in the feature map according to the downsampling scale;
[0060] Step S130, determining a first distance between the candidate frame and the original marked frame, and determining the candidate frame as a valid candidate frame when the first distance satisfies a preset first condition;
[0061] Step S140, determining the intersection-over-union ratio between the valid candidate box and the original marked box, and when the intersection-over-union ratio meets a preset second condition, determining that the valid candidate box is a positive sample;
[0062] Step S150, generating a token based on the positive sample, and calculating the self-attention weight based on the token.
[0063] It should be noted that the feature map can be a feature map that is rolled out by a convolution kernel after the input image is input into the neural network. For example, it can be obtained by multiplying the input image by a pre-set convolution kernel, and different convolution kernels will extract different feature maps. This embodiment does not limit the specific extraction process of the feature map. Those skilled in the art can adjust the convolution kernel and the neural network used according to actual needs, and no further limitations are made here.
[0064] It should be noted that, for candidate boxes that are far away from the original marked box, the value obtained when calculating the self-attention weight is actually close to 0, which can neither provide weights for the self-attention mechanism nor consume computational effort. Therefore, this part of the calculation can be considered as invalid calculation. Through the technical solution of this embodiment, the adaptive positive and negative sample selection algorithm (Adaptive Training Sample Selection, ATSS) can be used to adaptively select positive samples, thereby proving that all positive samples are around the original marked box, eliminating the above-mentioned invalid calculations, effectively reducing the amount of calculation, and improving the efficiency of image processing.
[0065] It should be noted that token is a computer term. In computer authentication, it means a token (temporary). In lexical analysis, it means a mark, which is an object representing the right to perform certain operations.
[0066] It should be noted that the Transformer model is a brand-new model proposed by Google in the paper "Attention Is All You Need". It abandons the convolutional neural network and recurrent neural network used in previous deep learning tasks. The Transformer model also adopts an encoder-decoder structure. The encoder part in the paper consists of 6 encoders stacked together, and the decoder part is the same. This model is widely used in the field of natural language processing (NLP), such as machine translation, question-answering systems, text summarization, and speech recognition. The Transformer model based on the self-attention mechanism is currently the most advanced neural network architecture, which includes an encoder and a decoder. The encoder is responsible for extracting the feature information of the text, extracting a feature vector for each word in the text, and thus obtaining the feature vector of the entire text. The decoder is responsible for using the feature vector extracted by the encoder to generate keywords that match the feature information as output.
[0067] It should be noted that the original marking box in the feature map can be obtained by pre-annotation. For example, after obtaining the sample set for training, the area where the target image exists in each feature map of the sample set is manually annotated to form the original marking box. Those skilled in the art are familiar with how to annotate the original marking box in the feature map, and will not go into details here.
[0068] It is worth noting that the first condition and the second condition can be in the form of a threshold or in the form of a ratio. For example, the first condition is the first threshold, and when the first distance is less than or equal to the first threshold, it is determined to be a valid candidate box; for another example, the second condition is the ratio threshold, and when the intersection and union ratio is greater than or equal to the ratio threshold, the valid candidate box is determined to be a positive sample. Technical personnel in this field can select appropriate first and second conditions according to actual needs, and can implement the ATSS technology, and make the obtained positive sample be an area with the target image.
[0069] It is understood by those skilled in the art that after obtaining the original marked frame, in order to achieve target detection, multiple candidate frames are used as anchor points in the feature map according to the downsampling scale, for example Figure 7 As shown, in Figure 7 There is a target image 730 in the feature map 700, which is pre-annotated with the original marking box 710. With 32×32 as the downsampling scale, 9 candidate boxes 720 are generated. It should be noted that the candidate box does not necessarily completely cover the feature map, because the candidate box is obtained according to the downsampling scale, and the size of the downsampling scale is not necessarily a multiple of the size of the feature map. Therefore, after the candidate box is generated, the edge of the outermost candidate box can have a certain distance from the edge of the feature map, which does not affect the target detection. No further details will be given here. It should be noted that the downsampling scale can be any value, such as the common 32×32, 16×16, 8×8. The specific value can be selected according to the timing requirements, and no further restrictions will be made here.
[0070] It should be noted that the intersection-over-union ratio is a commonly used parameter in target detection, which is usually used to describe the ratio of the intersection area to the union area between the candidate box and the original marked box. In image detection, it can be determined by the overlapping area. Those skilled in the art are familiar with how to calculate the intersection-over-union ratio between the candidate box and the original marked box, and will not be elaborated here.
[0071] It is worth noting that after determining the positive samples, all positive samples can be determined as tokens, for example Figure 7 As shown, in Figure 7 The positive samples determined in the feature map 700 include candidate boxes 720 numbered 5, 6, 8 and 9. The above four candidate boxes 720 are determined as tokens for calculating the self-attention weights, and the remaining candidate boxes 720 are determined to be invalid and not included in the calculation, which effectively reduces the amount of calculation.
[0072] Those skilled in the art can understand that the calculation expression of the self-attention weight is as follows: Among them, Q is the number of columns where the positive sample is located, K is the number of rows where the positive sample is located, T is the number of samples, and V is the feature map. At present, the feature map is cut into several blocks. Taking the sampling scale of 32×32 as an example, it is cut into 32×32 blocks according to 1 / 32 of the width and height of the feature map. According to the above formula, if the token is not cropped, 32×32 calculations are required, that is, there will be 32 square calculations. Since the original marking box will be annotated on the feature map during the training process, the position of the target image can be determined, and the calculation results of Q and K far away from the target image are almost 0. Based on this, the candidate boxes of the feature map are screened before calculating the self-attention weights to realize the function of ATSS. Through ATSS, the intimacy and distance are obtained, thereby shielding part of the QK calculation, which is conducive to speeding up the training speed and reducing unnecessary data noise. It is worth noting that in the case of tokens, those skilled in the art are familiar with how to calculate the self-attention weights. The specific calculation method does not belong to the improvement made in this embodiment, and will not be elaborated here.
[0073] In addition, refer to Figure 2 , in one embodiment, Figure 1 The step S130 shown also includes but is not limited to the following steps:
[0074] Step S210, obtaining a first coordinate and a second coordinate, wherein the first coordinate is the coordinate of the center point of the candidate frame, and the second coordinate is the coordinate of the center point of the original marked frame;
[0075] Step S220: determining the distance between the first coordinate and the second coordinate as the first distance.
[0076] It should be noted that the distance between the candidate box and the original marked box can be expressed by the distance between the center points. For example, a coordinate system can be established after obtaining the feature map, and the coordinates of the center point of the original marked box are obtained as the first coordinate, and the coordinates of the center point of the candidate box are obtained as the second coordinate. For example, the first coordinate is (x1, y1), and the second coordinate is (x2, y2). Then the first distance D can be calculated by the following formula:
[0077] In addition, refer to Figure 3 , in one embodiment, Figure 1 The step S140 shown also includes but is not limited to the following steps:
[0078] Step S310, determining a first area, where the first area is an intersection area of a valid candidate box and an original marked box;
[0079] Step S320, determining a second region, where the second region is a union region of the valid candidate frame and the original marked frame;
[0080] Step S330: determine the ratio of the area of the second region to the area of the first region as an intersection-over-union ratio.
[0081] It should be noted that, since the valid candidate box is determined by comparing the first distance with the first condition, it is possible to first determine whether the valid candidate box has an overlapping part with the original marked box. If the two do not overlap, the intersection and union ratio can be directly determined to be 0, thus eliminating unnecessary calculations.
[0082] It should be noted that the first area is the part of the valid candidate box and the original marked box, and is therefore equivalent to the intersection area between the valid candidate box and the original marked box. Similarly, the second area is the union of the valid candidate box and the original marked box. In order to simplify the calculation of the intersection ratio, it can be calculated by area. The calculation process can be performed according to the coordinate system obtained in the above embodiment, for example Figure 7 In the feature map shown, taking the valid candidate box numbered 5 as an example, the coordinates of the lower right corner of the valid candidate box (x3, y3), the coordinates of the point where the original marked box intersects the lower side of the valid candidate box (x4, y3), and the coordinates of the point where the original marked box intersects the upper side of the valid candidate box (x3, y4) are determined according to the established coordinate system. It can be determined that the area of the first region is S1 = |(x3-x4)×(y3-y4)|, and since the original marked box is obtained by annotation, its area S2 is known, and the area of the valid candidate box can be determined by the downsampling scale. For example, the downsampling scale is 32×32, and its area S3 = 32×32. Therefore, the area of the second region S4 = S2+S3-S1 can be calculated, and therefore the intersection-over-union ratio = S1 / S4 can be determined.
[0083] In addition, refer to Figure 4 , in one embodiment, Figure 1 The step S120 shown also includes but is not limited to the following steps:
[0084] Step S410, determining the current sampling depth of the feature map as the target sampling depth;
[0085] Step S420, obtaining a preset scale mapping table, where the scale mapping table records a mapping relationship between a sampling depth and a downsampling scale;
[0086] Step S430: determining a downsampling scale from a scale mapping table according to the target sampling depth.
[0087] It should be noted that in the target detection process, the input feature map needs to be downsampled multiple times, and the number of downsampling can be adjusted according to actual needs. The more downsampling times, the deeper the sampling depth, and the larger the box area mapped to the feature map, that is, the larger the area of the corresponding feature map. Objects with larger areas, in this case, if the downsampling scale is kept too large, the obtained token will be coarse and contain more invalid candidate boxes, which will increase the amount of calculation. Therefore, in order to reduce the amount of calculation and realize multi-scale cropping of tokens, the scale mapping table can be pre-set to match different downsampling scales according to the sampling depth. For example, for the first downsampling, a downsampling scale of 32×32 can be used, and for the second downsampling, a downsampling scale of 16×16 can be used. The specific values in the scale mapping table can be adjusted according to actual needs.
[0088] In addition, refer to Figure 5 , in one embodiment, Figure 1 The step S120 shown also includes but is not limited to the following steps:
[0089] Step S510, obtaining the sampling width and sampling length according to the downsampling scale
[0090] Step S520: Cut out at least two candidate boxes from the feature map according to the sampling width and the sampling length.
[0091] It should be noted that the downsampling scale is usually equal in length and width, such as the common 32×32, 16×16, 8×8. The specific value can be selected according to the actual needs of the feature map. When the original marked box marked in the feature map is small, a smaller downsampling scale can be used, so that more candidate boxes can be cut out in the feature map to ensure the accuracy of the calculation. Of course, it can also be determined according to the size of the feature map. For example, if at least two candidate boxes need to be cut out, the sampling width and sampling length are at most half of the size of the feature map. The specific cutting method is a technology well known to those skilled in the art and will not be elaborated here.
[0092] In addition, in one embodiment, the scale mapping table includes at least a first downsampling scale and a second downsampling scale, wherein the second downsampling scale is larger than the first downsampling scale, and a sampling depth corresponding to the second downsampling scale is larger than a sampling depth corresponding to the first downsampling scale.
[0093] It should be noted that the scale mapping table may include any number of downsampling scales, and each downsampling scale may correspond to any number of sampling depths. For example, the downsampling of each layer may correspond to a different downsampling scale, or the downsampling of several consecutive layers may correspond to the same downsampling scale. By making the sampling scale optional, the token cropping can be achieved at multiple scales, and the sampling scale can be adjusted according to the size of the target, so that the coarseness of the token is more in line with the requirements, the accuracy of the calculation is improved, and the amount of invalid calculation is reduced.
[0094] In addition, refer to Figure 6 In one embodiment, after executing Figure 1 After step S120, the following steps are also included but not limited to:
[0095] Step S610: When there is no candidate box whose first distance satisfies the first condition, and / or there is no candidate box whose intersection-over-union ratio satisfies the second condition, the sampling depth of the feature map is increased.
[0096] It should be noted that after obtaining multiple candidate boxes through downsampling scale, it is very likely that the downsampling scale is inappropriate and no qualified positive samples can be obtained. Therefore, when there is no first distance that satisfies the first condition, there is no valid candidate box. Even if the intersection-over-union ratio satisfies the second condition, the calculated self-attention weight will be close to 0, which is an invalid calculation. Therefore, when all candidate boxes do not have a first distance that satisfies the first condition, it can be determined that the downsampling scale of this downsampling is inappropriate, and it can be moved to a deeper feature map for token cropping, that is, the sampling depth of the feature map is increased. Since the downsampling scale can be reduced after the depth is increased, finer tokens can be obtained, so the cropped positive samples can be ensured to be usable. Of course, there is no intersection-over-union ratio that satisfies the second condition, which is similarly obtained and will not be repeated here.
[0097] It should be noted that after increasing the depth, the technical solution of this embodiment can be re-executed according to the new downsampling scale, which will not be elaborated here.
[0098] In addition, refer to Figure 8 The embodiment of the present invention further provides an image processing device based on ATSS. The image processing device based on ATSS 800 includes:
[0099] An acquisition unit 810 is used to acquire a feature map, wherein the feature map is pre-marked with an original marking frame;
[0100] A sampling unit 820 is used to obtain a downsampling scale of the feature map, and generate multiple candidate boxes in the feature map according to the downsampling scale;
[0101] A first determining unit 830 is used to determine a first distance between the candidate frame and the original marked frame, and when the first distance satisfies a preset first condition, determine that the candidate frame is a valid candidate frame;
[0102] A second determination unit 840 is used to determine an intersection-over-union ratio between the valid candidate frame and the original marked frame, and when the intersection-over-union ratio satisfies a preset second condition, the valid candidate frame is determined to be a positive sample;
[0103] The calculation unit 850 is used to generate a token according to the positive sample and calculate the self-attention weight according to the token.
[0104] In addition, refer to Fig. 9 An embodiment of the present invention further provides an electronic device, the electronic device 900 comprising: a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.
[0105] The processor 920 and the memory 910 may be connected via a bus or in other ways.
[0106] The non-transitory software program and instructions required to implement the ATSS-based image processing method of the above embodiment are stored in the memory 910. When executed by the processor 920, the ATSS-based image processing method applied to the electronic device in the above embodiment is executed, for example, the above-described Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Method step S610 in .
[0107] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0108] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions, which are executed by a processor or controller, for example, by a processor in the above-mentioned electronic device embodiment, so that the above-mentioned processor can execute the ATSS-based image processing method applied to the electronic device in the above-mentioned embodiment, for example, to execute the above-described Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Method step S610 in. It will be appreciated by those skilled in the art that all or some of the steps and devices in the above disclosed method can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium, which can include a computer storage medium (or a non-transitory storage medium) and a communication storage medium (or a temporary storage medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other storage medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication storage media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery storage media.
[0109] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld electronic devices or portable electronic devices, tablet electronic devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or electronic devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing electronic devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage electronic devices.
[0110] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. An image processing method based on an adaptive positive and negative sample selection algorithm, characterized in that: The method comprises: Acquire a feature map, wherein the feature map is pre-marked with an original marking frame; Obtaining a downsampling scale of the feature map, and generating a plurality of candidate boxes in the feature map according to the downsampling scale; Determine a first distance between the candidate frame and the original marked frame, and when the first distance satisfies a preset first condition, determine that the candidate frame is a valid candidate frame; Determine an intersection-and-union ratio between the valid candidate frame and the original marked frame, and when the intersection-and-union ratio satisfies a preset second condition, determine that the valid candidate frame is a positive sample; Generate a token according to the positive sample, and calculate the self-attention weight according to the token; The determining the intersection-over-union ratio between the valid candidate frame and the original marked frame includes: Determine a first area, where the first area is an intersection area of the valid candidate box and the original marked box; Determine a second area, where the second area is a union area of the valid candidate box and the original marked box; Determine the area S2 of the original marked frame, determine the area S3 of the valid candidate frame based on the downsampling scale, and determine the area S4 of the second region based on the area S2 of the original marked frame, the area S1 of the first region, and the area S3 of the valid candidate frame, wherein S4=S2+S3-S1; Determine a ratio S1 / S4 of the area of the second region to the area of the first region as the intersection-over-union ratio; The determining a first distance between the candidate frame and the original marked frame includes: Obtaining a first coordinate and a second coordinate, wherein the first coordinate is the coordinate of the center point of the candidate frame, and the second coordinate is the coordinate of the center point of the original marked frame; determining a distance between the first coordinate and the second coordinate as the first distance; After generating a plurality of candidate boxes in the feature map according to the downsampling scale, the method further includes: When there is no candidate box whose first distance satisfies the first condition, and / or there is no candidate box whose intersection-over-union ratio satisfies the second condition, the sampling depth of the feature map is increased.
2. The image processing method based on the adaptive positive and negative sample selection algorithm according to claim 1, characterized in that: The obtaining of the downsampling scale of the feature map comprises: Determine the current sampling depth of the feature map as a target sampling depth; Acquire a preset scale mapping table, wherein the scale mapping table records a mapping relationship between a sampling depth and the downsampling scale; The downsampling scale is determined from the scale mapping table according to the target sampling depth.
3. The image processing method based on the adaptive positive and negative sample selection algorithm according to claim 2, characterized in that: Generating a plurality of candidate frames in the feature map according to the downsampling scale includes: The sampling width and sampling length are obtained according to the downsampling scale At least two candidate boxes are cut out from the feature map according to the sampling width and the sampling length.
4. The image processing method based on the adaptive positive and negative sample selection algorithm according to claim 2, characterized in that: The scale mapping table includes at least a first downsampling scale and a second downsampling scale, wherein the second downsampling scale is larger than the first downsampling scale, and the sampling depth corresponding to the second downsampling scale is larger than the sampling depth corresponding to the first downsampling scale.
5. An image processing device based on an adaptive positive and negative sample selection algorithm, characterized in that: include: An acquisition unit, used for acquiring a feature map, wherein the feature map is pre-marked with an original marking frame; A sampling unit, used to obtain a downsampling scale of the feature map, and generate a plurality of candidate boxes in the feature map according to the downsampling scale; A first determining unit, configured to determine a first distance between the candidate frame and the original marked frame, and determine that the candidate frame is a valid candidate frame when the first distance satisfies a preset first condition; A second determination unit is used to determine an intersection-over-union ratio between the valid candidate frame and the original marked frame, and when the intersection-over-union ratio satisfies a preset second condition, determine that the valid candidate frame is a positive sample; A calculation unit, used to generate a token according to the positive sample, and calculate a self-attention weight according to the token; The determining the intersection-over-union ratio between the valid candidate frame and the original marked frame includes: Determine a first area, where the first area is an intersection area of the valid candidate box and the original marked box; Determine a second area, where the second area is a union area of the valid candidate box and the original marked box; Determine the area S2 of the original marked frame, determine the area S3 of the valid candidate frame based on the downsampling scale, and determine the area S4 of the second region based on the area S2 of the original marked frame, the area S1 of the first region, and the area S3 of the valid candidate frame, wherein S4=S2+S3-S1; Determine a ratio S1 / S4 of the area of the second region to the area of the first region as the intersection-over-union ratio; The determining a first distance between the candidate frame and the original marked frame includes: Obtaining a first coordinate and a second coordinate, wherein the first coordinate is the coordinate of the center point of the candidate frame, and the second coordinate is the coordinate of the center point of the original marked frame; determining a distance between the first coordinate and the second coordinate as the first distance; After generating a plurality of candidate frames in the feature map according to the downsampling scale, the method further includes: When there is no candidate box whose first distance satisfies the first condition, and / or there is no candidate box whose intersection-over-union ratio satisfies the second condition, the sampling depth of the feature map is increased.
6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method based on the adaptive positive and negative sample selection algorithm as described in any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the image processing method based on the adaptive positive and negative sample selection algorithm as described in any one of claims 1 to 4.
