Detection model training method, device, storage medium and computer equipment

Abstract

The present application relates to a detection model training method, device, storage medium and computer equipment, the method comprising: obtaining an original image to be processed and more than one type of labeled images; for each type of labeled images, randomly selecting a target position from a target area of ​​the original image as an embedding position of the labeled image; for each type of labeled image, embedding at least a part of the labeled image into the original image according to the corresponding embedding position to obtain a corresponding sample image; using the sample image as a training sample, and using the label category of the labeled image embedded in the sample image as a corresponding training label; training the detection model to be trained through the training sample and the corresponding training label. The solution provided by the present application can improve the training efficiency.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a detection model training method, apparatus, computer-readable storage medium, and computer equipment. Background Art

[0002] With the development of computer technology, machine learning technology has emerged. Through machine learning technology, computers can be trained to simulate or realize human learning behavior, thus bringing convenience to people's lives and work. For example, in the field of image processing, models can be trained through training data so that the models can learn the ability to classify or locate, so that machines can replace human labor to process images.

[0003] In practical applications, for example, when a detection model that can identify a target object (such as a watermark or trademark) needs to be trained, a large amount of labeled data is often required for model training. However, the traditional method usually consumes manpower to label the category of the target object and the location of the target object in the image. The labeling speed is slow, resulting in low model training efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a detection model training method, apparatus, computer-readable storage medium and computer equipment to address the technical problem of low model training efficiency caused by manual labeling of data.

[0005] A detection model training method, comprising:

[0006] Obtaining an original image to be processed and labeled images of more than one category;

[0007] For each type of labeled image, randomly selecting a target position from the target area of ​​the original image as the embedding position of the labeled image;

[0008] For each type of marked image, embedding at least a portion of the marked image into the original image according to the corresponding embedding position to obtain a corresponding sample image;

[0009] Using the sample image as a training sample, and using the label category of the label image embedded in the sample image as a corresponding training label;

[0010] The detection model to be trained is trained using the training samples and corresponding training labels.

[0011] A detection model training device, the device comprising:

[0012] An acquisition module, used for acquiring an original image to be processed and more than one type of labeled images;

[0013] A selection module, for randomly selecting a target position from a target area of ​​the original image as an embedding position of the marked image for each type of marked image;

[0014] An embedding module, for embedding at least a portion of each type of labeled image into the original image according to corresponding embedding positions to obtain a corresponding sample image;

[0015] A determination module, taking the sample image as a training sample, and taking the tag category of the tag image embedded in the sample image as a corresponding training tag;

[0016] The training module is used to train the detection model to be trained through the training samples and corresponding training labels.

[0017] In one of the embodiments, the acquisition module is also used to acquire the original image to be processed and more than one type of marking templates; randomly select a target size ratio from a preset size ratio range; and scale each type of marking template according to the target size ratio based on the size of the original image to obtain a corresponding marking image.

[0018] In one embodiment, the target position includes a core position and a non-core position; the selection module is also used to determine the core position in the target area of ​​the original image; obtain the probability value when the core position is used as the embedding position; the probability value corresponding to the core position is the maximum value of the probability values ​​corresponding to each target position in the target area; according to the distance between each non-core position in the target area and the core position, determine the probability value when each non-core position is used as the embedding position; the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position; for each type of marked image, select the corresponding target position as the embedding position of the marked image according to the probability value corresponding to each target position in the target area of ​​the original image.

[0019] In one of the embodiments, the selection module is also used to obtain a preset number of platform-specific images; each of the platform-specific images includes a marker image corresponding to the corresponding platform; based on the coordinates of the target vertices of each marker image in the platform-specific image, the average coordinates corresponding to the target vertex are determined; and the average coordinates are used as the core position in the target area of ​​the original image.

[0020] In one embodiment, the target area includes an upper left corner area and a lower right corner area; the selection module is also used to, when the target area is the upper left corner area, for each type of marking template, select the corresponding target position as the embedding position corresponding to the upper left vertex of the marking image according to the probability values ​​corresponding to each target position in the upper left corner area of ​​the original image; when the target area is the lower right corner area, for each type of marking template, select the corresponding target position as the embedding position corresponding to the lower right vertex of the marking image according to the probability values ​​corresponding to each target position in the lower right corner area of ​​the original image.

[0021] In one embodiment, the sample image includes a first sample image and a second sample image; the embedding module is also used to determine, for each type of labeled image, a first labeled image to be completely embedded and a second labeled image to be occluded and embedded in the labeled image; the first labeled image is completely embedded into the original image according to a corresponding embedding position to obtain a corresponding first sample image; the second labeled image is completely embedded into the original image according to a corresponding embedding position, and a target occlusion ratio is randomly selected from a preset occlusion ratio range, and a portion of the second labeled image is moved out to the boundary of the original image according to the target occlusion ratio to obtain a corresponding second sample image.

[0022] In one of the embodiments, the determination module is also used to determine the tag category of the tag image embedded in each of the sample images, and the position information of the tag image in the original image; the sample image is used as a training sample, and the tag category and corresponding position information of the tag image embedded in the sample image are used together as training labels of the training sample.

[0023] In one of the embodiments, the training module is also used to crop the sample image according to the target area to obtain corresponding sample image blocks; perform feature extraction on each of the sample image blocks through the detection model to be trained to obtain a corresponding feature map, and perform detection and output prediction results based on the feature map; adjust the model parameters of the detection model according to the difference between the prediction results corresponding to the sample image blocks and the corresponding training labels, and stop the training until the training stop condition is met.

[0024] In one embodiment, the device also includes a marker detection module, which is used to obtain a video to be detected and a trained detection model; extract a preset number of video frames from the video to be detected, and crop each of the video frames according to the target area to obtain a corresponding target image block; input each of the target image blocks into the trained detection model respectively, and output the detection results corresponding to each of the target image blocks; and fuse the detection results of each target image block to obtain a detection result corresponding to the video to be detected.

[0025] In one embodiment, the marker detection module is also used to input each of the target image blocks into the trained detection model respectively; the input target image blocks are processed in turn by at least three groups of convolution groups in the trained detection model; wherein the downsampling layer in the last group of convolution groups is a hole convolution with a step size of a preset value to keep the size of the feature map output by the last group of convolution groups at a preset size; the feature map output by the convolution group of the middle group is convolved to obtain a first feature map to be detected; the feature map output by the last group of convolution groups is used as a second feature map to be detected; the second feature map to be detected is convolved to obtain at least one third feature map to be detected; the first feature map to be detected, the second feature map to be detected and the third feature map to be detected are respectively detected to obtain the corresponding candidate detection results and the confidence corresponding to the candidate results; from the candidate detection results corresponding to each feature map to be detected, the candidate detection results whose corresponding confidences meet the high confidence conditions are screened out as the detection results corresponding to the input target image block.

[0026] In one embodiment, the detection result includes the tag category to which the marked image in the video to be detected belongs; the device also includes a video push module for obtaining a video filtering instruction; the video filtering instruction includes a first target category; the tag category corresponding to each video in the video library is determined by the trained detection model; the video library is searched for videos to be filtered whose tag category is the first target category; in response to the video filtering instruction, the videos in the video library except the video to be filtered are pushed to the user terminal that initiates the video filtering instruction.

[0027] In one of the embodiments, the video push module is also used to obtain a video search instruction; the video search instruction includes a second target category; determine the label category corresponding to each video in the video library through the trained detection model; search for target videos with a label category of the second target category from the video library; and in response to the video search instruction, push the target video to the user terminal that initiated the video instruction.

[0028] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0029] Obtaining an original image to be processed and labeled images of more than one category;

[0030] For each type of labeled image, randomly selecting a target position from the target area of ​​the original image as the embedding position of the labeled image;

[0031] For each type of marked image, embedding at least a portion of the marked image into the original image according to the corresponding embedding position to obtain a corresponding sample image;

[0032] Using the sample image as a training sample, and using the label category of the label image embedded in the sample image as a corresponding training label;

[0033] The detection model to be trained is trained using the training samples and corresponding training labels.

[0034] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0035] Obtaining an original image to be processed and labeled images of more than one category;

[0036] For each type of labeled image, randomly selecting a target position from the target area of ​​the original image as the embedding position of the labeled image;

[0037] For each type of marked image, embedding at least a portion of the marked image into the original image according to the corresponding embedding position to obtain a corresponding sample image;

[0038] Using the sample image as a training sample, and using the label category of the label image embedded in the sample image as a corresponding training label;

[0039] The detection model to be trained is trained using the training samples and corresponding training labels.

[0040] The above-mentioned detection model training method, device, computer-readable storage medium and computer equipment randomly embed each type of labeled image into the original image, and when embedding, simulate the situation that the labeled image may be edited or compressed in the real scene, with full embedding and partial embedding, so that labeled training data can be automatically generated for training the detection model. Among them, the training label in the training data is the label category to which the embedded labeled image belongs. In this way, there is no need to manually label the training data, and a variety of random strategies are used to simulate the situation where the real labeled image appears in the original image, which greatly reduces the labor cost and labeling efficiency of labeling training data, thereby greatly improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An application environment diagram of a detection model training method in an embodiment;

[0042] Figure 2 A schematic diagram of a flow chart of a detection model training method in one embodiment;

[0043] Figure 3 A schematic diagram of each marking template in an embodiment of an embodiment;

[0044] Figure 4 A flowchart of a step of randomly selecting a target position from a target area of ​​an original image as an embedding position of a marked image for each type of marked image in one embodiment;

[0045] Figure 5 A schematic flow chart of steps of performing labeling detection on a video to be detected by using a trained detection model in one embodiment;

[0046] Figure 6 A schematic diagram of a network structure of a detection network based on an SSD algorithm in one embodiment;

[0047] Figure 7 A schematic diagram of the network structure of a RetinaNet network in one embodiment;

[0048] Figure 8 It is a flowchart of the steps of performing labeling detection on a video to be detected by using a trained detection model and obtaining a detection result in a specific embodiment;

[0049] Fig. 9 It is a structural block diagram of a detection model training device in one embodiment;

[0050] Fig.10 It is a structural block diagram of a detection model training device in another embodiment;

[0051] Fig.11FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Figure 1 FIG. 1 is an application environment diagram of a detection model training method in an embodiment. Figure 1 , the detection model training method is applied to the detection model training system. The detection model training system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The terminal 110 can specifically be a desktop terminal or a mobile terminal, and the mobile terminal can specifically be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented with an independent server or a server cluster composed of multiple servers. Both the terminal 110 and the server 120 can be used alone to execute the detection model training method provided in the embodiment of the present application. The terminal 110 and the server 120 can also be used in collaboration to execute the detection model training method provided in the embodiment of the present application.

[0054] It should be noted that the detection model training method involves machine learning (ML), which is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0055] like Figure 2 As shown, in one embodiment, a detection model training method is provided. This embodiment mainly uses the method applied to a computer device as an example, and the computer device can specifically be the above-mentioned Figure 1 The terminal 110 or the server 120 in FIG. Figure 2 , the detection model training method specifically includes the following steps:

[0056] S202, obtaining an original image to be processed and more than one type of labeled images.

[0057] The original image is an image to be processed without marking information. The original image can be a real image captured by a camera, or a certain frame or multiple frames of video segments from a video file, or an image synthesized by a computer device. Marking information is information used for proprietary marking, and can be specific text, icons, signs, or audio information with a certain degree of recognition. A marked image is an image that includes marking information, and can be a watermark image or a trademark image.

[0058] Specifically, the computer device can obtain one or more original images from the local or other computer devices, where "multiple" means more than one. The quantifiers "multiple", "multiple", or "multiple categories" mentioned in the embodiments of the present application, unless otherwise specified, all mean "more than one", "more than one", or "more than one category", etc. The computer device can pre-acquire tag information belonging to different tag categories, and generate corresponding tagged images according to the corresponding tag information. Among them, the tag category can specifically be a category used to mark different platforms, that is, the category of the platform to which the tagged image belongs. Different platforms can specifically be different media platforms, such as the "Tik Tok" platform, the "Weishi" platform, the "Xigua Video" platform, the "Kuaishou" platform, the "Huoshan Short Video" platform, or the "Pipixia" platform. The tagged images corresponding to different platforms have the characteristics of their respective platforms and are different from each other.

[0059] In one embodiment, in order to ensure the training effect of the detection model, the computer device may adjust the format of the acquired original image to obtain an original image of a uniform format and size.

[0060] In one embodiment, the computer device may also obtain a marking template having RGBA (red green blue Alpha) four-channel information, and adjust the size of the marking template according to a preset size ratio based on the size of the original image to obtain a corresponding marking image.

[0061] S204: For each type of labeled image, randomly select a target position from the target area of ​​the original image as an embedding position of the labeled image.

[0062] The target area is the area used to embed the marker image, which can be the entire area of ​​the original image or a partial area of ​​the original image, such as the upper left corner area, lower right corner area, lower left corner area or upper right corner area of ​​the original image with a preset size. The embedding position is a position point used to locate the specific position of the marker image in the original image.

[0063] Specifically, the computer device can randomly select a certain type of labeled image from various types of labeled images, and randomly select a target position from the target area of ​​the original image as the embedding position of the labeled image. It can be understood that in order to enable the detection model to obtain a good detection effect and accurately distinguish different labeled categories, when constructing training data, an equal number of sample images are constructed for each type of labeled image, that is, when the computer device selects the labeled image, the probability of each type of labeled image being selected is equal. Among them, the training data specifically includes training samples and training labels.

[0064] In one embodiment, the computer device may pre-set a fixed area as the target area of ​​the original image. Alternatively, the computer device may determine the target area based on the area where the tagged image is located in the real image including the tagged image in actual situations. For example, the computer device may count the locations where the tagged image appears in a large number of real images, and use the area where the probability of the tagged image appearing in the real image is greater than a preset probability threshold or a larger area as the target area. The real image here is an image with platform tag information corresponding to different platforms in the real scene, which can also be called a platform-specific image.

[0065] In one embodiment, when the computer device selects a target position from the target area, the probability values ​​of each position in the target area being selected can be set to the same probability value or different probability values. In one embodiment, the computer device can determine a position in the target area as a core position, and the core position corresponds to the largest probability value of selection, and the probability values ​​of selection corresponding to other non-core positions decrease from the core position to the surrounding areas.

[0066] In one embodiment, the computer device may use the target position selected in the target area as the embedding position of the mark image, specifically as the embedding position corresponding to the upper left vertex of the mark image, as the embedding position corresponding to the lower right vertex of the mark image, or as the embedding position corresponding to the center point of the mark image, etc., which is not limited in this embodiment of the present application.

[0067] S206 , for each type of labeled image, embed at least a portion of the labeled image into the original image according to the corresponding embedding position to obtain a corresponding sample image.

[0068] Specifically, after determining the embedding position corresponding to a certain marked image, the computer device may embed at least a portion of the image content in the marked image into the original image according to the embedding position to obtain a corresponding sample image.

[0069] In one embodiment, the computer device may paste the entire marked image onto the original image according to the embedding position to obtain a corresponding sample image.

[0070] In one embodiment, the computer device can paste the entire tagged image to the original image according to the embedding position, and then move the tagged image toward the edge of the original image, so that a portion of the tagged image is moved out to the edge of the original image, and a portion of the image content is retained on the original image to obtain a corresponding sample image. This can better simulate the actual situation in which the edge of the tagged image may be clipped.

[0071] In one embodiment, the computer device may crop a portion of the image content from the tagged image, and paste the portion of the tagged image into the original image according to the embedding position to obtain a corresponding sample image.

[0072] For example, the computer device may pre-set an occlusion probability value, such as 0.25, which means that the labeled images in 25% of the training samples are occluded, and the labeled images in 75% of the training samples are complete. For training samples where the labeled images are occluded, the computer device may randomly select a target occlusion ratio from a preset occlusion ratio range to adjust the height, width or area of ​​the labeled image in the original image according to the target occlusion ratio. For example, the computer device may set a preset occlusion ratio range of 0.3 to 0.7, and the computer device may evenly select a target occlusion ratio from 0.3 to 0.7.

[0073] S208: Use the sample image as a training sample, and use the tag category of the tag image embedded in the sample image as a corresponding training label.

[0074] Specifically, the computer device can randomly generate a large number of sample images including labeled images of different labeled categories based on the random method mentioned in the above steps. The embedding positions of the labeled images in these generated sample images are randomly distributed, and the coverage areas of the labeled images in the sample images are also randomly distributed. Then, the computer device can use these sample images as training samples, and use the labeled category of the labeled image embedded in each sample image as the training label corresponding to the corresponding training sample.

[0075] In one embodiment, step S208, i.e., taking the sample image as a training sample and taking the tag category of the tag image embedded in the sample image as the corresponding training label, specifically includes: determining the tag category of the tag image embedded in each sample image and the position information of the tag image in the original image; taking the sample image as a training sample, and taking the tag category and the corresponding position information of the tag image embedded in the sample image as the training label of the training sample.

[0076] In one embodiment, the computer device can determine the tag category of the tag image embedded in each sample image, and the position information of the tag image in the original image. The position information of the tag image in the original image is used to locate the tag image, which can be the coordinates of the upper left vertex and the lower right vertex of the embedded tag image, or the coordinates of the lower left vertex and the upper right vertex, etc. Furthermore, the computer device can use the sample image as a training sample, and use the tag category and the corresponding position information of the tag image embedded in the sample image as the training label of the training sample. In this way, the detection model trained by the corresponding training samples and training labels can not only predict the tag category of the image to be detected, but also locate the tag image in the image to be detected.

[0077] S210, training the detection model to be trained using training samples and corresponding training labels.

[0078] The detection model is a convolutional neural network model used for label detection of an image to be detected or a video to be detected. When the training label is only the label category, the trained detection model can be used to classify the label image included in the input image to be detected. When the training label includes the label category and the location information of the label image, the trained detection model can be used to classify and locate the label image included in the input image to be detected.

[0079] Specifically, the training of the detection model is a supervised training process. The computer device inputs the training sample into the detection model, takes the training label corresponding to the training sample as the target output, and adjusts the model parameters of the detection model so that the actual output of the detection model continuously approaches the target output.

[0080] In one embodiment, the computer device may input the training sample into the detection model for training to obtain a prediction result. A loss function is constructed according to the difference between the prediction result and the training label. The model parameters when the loss function is minimized are used as the model parameters of the detection model, and the step of inputting the training sample into the detection model for training to obtain the detection result is returned until the training stop condition is met and the training is stopped.

[0081] The training stop condition is a condition for ending the model training. The training stop condition may be reaching a preset number of iterations, or the performance index of the detection model after adjusting the model parameters reaches a preset index.

[0082] In one embodiment, the model structure of the detection model and the detection algorithm used are not limited in the embodiments of the present application. The detection model can be implemented by a two-stage detection algorithm based on a neural network, or by a one-stage detection algorithm based on a neural network. For example, a computer device can construct a detection model based on an SSD algorithm (Single Shot MultiBox Detector, a target detection algorithm) or a RetinaNet algorithm (a single-stage target detection algorithm) through a convolutional neural network.

[0083] In one embodiment, step S210, that is, the step of training the detection model to be trained through training samples and corresponding training labels, specifically includes: cropping the sample image according to the target area to obtain the corresponding sample image block; extracting features of each sample image block through the detection model to be trained to obtain a corresponding feature map, and performing detection and outputting prediction results based on the feature map; adjusting the model parameters of the detection model according to the difference between the prediction result corresponding to the sample image block and the corresponding training label, and stopping the training until the training stop condition is met.

[0084] In one embodiment, the computer device may crop the sample image according to the target area to obtain the corresponding sample image block. Then, the computer device may input each sample image block into the detection model to be trained, perform feature extraction on each sample image block through the convolution layer included in the detection model to obtain the corresponding feature map, and perform label detection based on each feature map to output the prediction result. When the training label is a label category, the corresponding prediction result is the predicted label category. When the training label is the label category and the position information of the label image in the target image block, the corresponding prediction result is the predicted label category and the regression box. The computer device may adjust the model parameters of the detection model in the direction of reducing the difference according to the difference between the prediction result corresponding to the sample image block and the corresponding training label, until the training is stopped when the training stop condition is met.

[0085] In the above embodiment, the sample image is cropped according to the target area to obtain the corresponding sample image block, and then the detection model to be trained is trained by the sample image block. This can greatly reduce the interference and additional processing caused by background information unrelated to the labeled image during the training process, thereby improving the efficiency and accuracy of model training.

[0086] It can be understood that in practical applications, in order to avoid the occlusion of the effective content of the original image by the marked image, the marked image usually appears in the upper left corner area or the lower right corner area of ​​the original image, which can not only play the marking effect, but also will not occlude the effective content of the original image. Based on this, when designing training data, if the marked image is only embedded in the fixed position of the upper left corner area and the lower right corner area of ​​the original image, the detection model may mistakenly think that the fixed position is very likely to have a marked image, but does not really have the ability to judge that different positions contain marked images, and only learns the ability to distinguish positions. Therefore, in step S204, some random factors are added when determining the embedding position of the marked image to simulate the real situation. In step S206, at least a part of the marked image is embedded in the original image, mainly considering that the marked image may be edited or compressed in the real scene, so that only a part of the marked image appears, and the addition of random occlusion factors enables the detection model to handle the situation where the marked image is only partially visible. In this way, by simulating the real generation of marked images through a variety of random strategies, reliable and effective training data with annotation information is automatically generated, which greatly reduces the manpower cost of obtaining training data. And the detection model is trained by using the training data generated above that fits the actual situation, so that the trained detection model has higher detection accuracy.

[0087] The above detection model training method randomly embeds each type of labeled image into the original image, and during embedding, simulates the situation in which the labeled image may be edited or compressed in real scenes, with full embedding and partial embedding, so that labeled training data can be automatically generated for training the detection model. Among them, the training label in the training data is the label category to which the embedded labeled image belongs. In this way, there is no need to manually label the training data, and a variety of random strategies are used to simulate the situation where the real labeled image appears in the original image, which greatly reduces the labor cost and labeling efficiency of labeling training data, thereby greatly improving the efficiency of model training.

[0088] In one embodiment, step S202, i.e., the step of obtaining the original image to be processed and more than one type of marked images, specifically includes: obtaining the original image to be processed and more than one type of marked templates; randomly selecting a target size ratio from a preset size ratio range; scaling each type of marked template according to the target size ratio based on the size of the original image to obtain a corresponding marked image.

[0089] Among them, the marking template is a marking sample corresponding to each platform, which can be used as a template. The size ratio is the size ratio between the sizes of two objects. In the embodiment of the present application, it specifically refers to the size ratio between the marked image and the original image. The preset size ratio range is a pre-set range consisting of a series of size ratios, such as a height ratio range, a width ratio range, or an area ratio range.

[0090] Specifically, the computer device can obtain the marking templates corresponding to different platforms, that is, the marking templates belonging to different marking categories. The computer device can specifically obtain the marking template with RGBA4 channel data. Then, the computer device can select the target size ratio from the preset size ratio range, and scale the various marking templates according to the target size ratio according to the size of the original image to obtain the corresponding marking image.

[0091] In one embodiment, the computer device may randomly select a marking template from the candidate marking templates, wherein the probability of selecting each marking template is the same. The computer device may pre-set a height ratio range between the marked image and the original image, such as a height ratio range of 0.04 to 0.14. The computer device may randomly select a target height ratio from the height ratio range, and then scale the marking template according to the corresponding target height ratio to obtain a marked image. It can be understood that the preset size ratio range may also be a preset width ratio range or area ratio range, etc., and the corresponding selected target size ratio may also be a target width ratio or a target area ratio, etc., which is not limited in the embodiments of the present application.

[0092] In one embodiment, for different size ratios in the preset size ratio range, the computer device can set corresponding selection probabilities. Then, the target size ratio is randomly selected from the preset size ratio range according to the corresponding selection probability. In one embodiment, the setting of the selection probability can be determined based on a real image containing a real tagged image. For size ratios that appear frequently, a larger selection probability is set, and for size ratios that appear rarely, a smaller selection probability is set. In this way, the size of the tagged image relative to the original image can be more realistically simulated to improve the training effect of the detection model.

[0093] refer to Figure 3 , Figure 3 Schematic diagram of each marking template in an embodiment. For example, the marking template may be a marking template corresponding to a platform such as "Weishi", "Guangguang Xiaoshou" or "Xiafanshou". It can be understood that Figure 3 The above are only some examples. The marking templates and marking images mentioned in the embodiments of the present application include but are not limited to the above-mentioned ones. For example, the marking template can also be the template corresponding to the "Tik Tok" platform, the "Xigua Video" platform, the "Kuaishou" platform, the "Huoshan Short Video" platform, or the "Pipixia" platform.

[0094] In one embodiment, for each type of label template, when the computer device constructs the corresponding training data, the frequency of using each type of label template can be the same or different, and the embodiment of the present application does not limit this. With the emergence of different platforms on the market, there will be more and more corresponding label templates. For newly emerging label templates, there is no need to annotate the training data through manpower. The training data corresponding to the label template can be constructed in the manner mentioned in the embodiment of the present application to retrain the detection model to improve the detection range and capability of the detection model.

[0095] In the above embodiment, a target size ratio is randomly selected from a preset size ratio range, and then each type of marking template is scaled according to the target size ratio based on the size of the original image. This can obtain marked images of different sizes, which are more in line with the actual situation, making the constructed training samples more accurate and greatly improving the training effect of the detection model.

[0096] In one embodiment, the target position includes a core position and a non-core position; step S202, that is, for each type of marked image, the step of randomly selecting a target position from the target area of ​​the original image as the embedding position of the marked image specifically includes the following steps:

[0097] S402, determining a core position in a target area of ​​the original image.

[0098] The core position is the position in the target area with the highest probability of being selected as the embedding position. The computer device can select the position with the highest number of embedding positions as the core position based on a preset number of image data of real scenes. Alternatively, the computer device can also select the average position of the embedding positions corresponding to the respective marked images as the core position based on multiple image data of real scenes.

[0099] In one embodiment, step S402, i.e., the step of determining the core position in the target area of ​​the original image, specifically includes: obtaining a preset number of platform-specific images; each platform-specific image includes a marker image corresponding to the corresponding platform; according to the coordinates of the target vertices of each marker image in the platform-specific image, respectively, determining the average coordinates corresponding to the target vertex; and using the average coordinates as the core position in the target area of ​​the original image.

[0100] The target vertex of the marked image may be the upper left vertex, the lower left vertex, the upper right vertex or the lower right vertex. When the target area is the upper left corner area of ​​the original image, the target vertex of the marked image may be the upper left vertex; when the target area is the lower right corner area of ​​the original image, the target vertex of the marked image may be the lower right vertex.

[0101] In one embodiment, the computer device can obtain a preset number of platform-specific images, each of which is an image in a real scene. The computer device can determine the vertex coordinates of the target vertices of each labeled image in the platform-specific image, calculate the average coordinates of each vertex coordinate, and then use the average coordinates as the core position in the target area of ​​the original image. It can be understood that in this way, according to the core position determined by the target vertex, when the core position is selected as the embedding position when constructing training data, the core position in the original image can be used as the position of the corresponding target vertex of the labeled image, thereby transferring the labeled image to the original image.

[0102] For example, a computer device can obtain 50 images of real scenes, that is, platform-specific images. Calculate the average coordinates of the corresponding marked images (such as watermark images) in the upper left corner area of ​​these 50 platform-specific images, and use the position corresponding to the average coordinates as the core position in the upper left corner area. Let the height of the original image be h and the width be w, and establish a rectangular coordinate system downward with the upper left vertex of the original image as the coordinate origin. The coordinates of the core position can be exemplarily (w*0.03, h*0.015). When the target area is the lower right corner area, the coordinates of the core position of the lower right corner area can be exemplarily (ww*0.03, hh*0.035).

[0103] In the above embodiment, the average position corresponding to the target vertices of the marked image in a preset number of platform-specific images in the real scene is used as the core position to allocate the maximum probability value, which is more in line with the real situation.

[0104] S404, obtaining a probability value when the core position is used as the embedding position; the probability value corresponding to the core position is the maximum value of the probability values ​​corresponding to the target positions in the target area.

[0105] Specifically, the computer device may pre-set a probability value when the core position is used as the embedding position, and the probability value corresponding to the core position is the maximum value of the probability values ​​corresponding to the target positions in the target area.

[0106] S406, determining the probability value of using each non-core position as the embedding position according to the distance between each non-core position and the core position in the target area; the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position.

[0107] Specifically, other positions in the target area except the core position can be called non-core positions. The computer device can calculate the distance between each non-core position and the core position in the target area, and the distance can be specifically represented by a paradigm distance. According to the distance between each non-core position and the core position in the target area, the probability value of each non-core position when used as an embedded position is determined, wherein the size of the probability value is inversely proportional to the paradigm distance from the core point. In other words, the probability distribution of each position decreases around the core position. It can be understood that the sum of the probability values ​​corresponding to each target position in the target area is 1.

[0108] S408 , for each type of labeled image, according to the probability values ​​corresponding to the target positions in the target area of ​​the original image, select the corresponding target positions as the embedding positions of the labeled image.

[0109] Specifically, for the labeled images selected from various types of labeled images, the computer device may select corresponding target positions as the embedding positions of the labeled images according to the probability values ​​corresponding to the target positions in the target area of ​​the original image.

[0110] In one embodiment, the target area includes an upper left corner area and a lower right corner area; for each type of labeled image, the corresponding target position is selected as the embedding position of the labeled image according to the probability value corresponding to each target position in the target area of ​​the original image, including: when the target area is the upper left corner area, for each type of labeling template, the corresponding target position is selected as the embedding position corresponding to the upper left vertex of the labeled image according to the probability value corresponding to each target position in the upper left corner area of ​​the original image; when the target area is the lower right corner area, for each type of labeling template, the corresponding target position is selected as the embedding position corresponding to the lower right vertex of the labeled image according to the probability value corresponding to each target position in the lower right corner area of ​​the original image.

[0111] In one embodiment, when the target area is the upper left corner area, the computer device can select the corresponding target position as the embedding position corresponding to the upper left vertex of the marked image according to the probability value corresponding to each target position in the upper left corner area of ​​the original image, and then embed the upper left corner of the marked image at this position when embedding the marked image. When the target area is the lower right corner area, the computer device can select the corresponding target position as the embedding position corresponding to the lower right vertex of the marked image according to the probability value corresponding to each target position in the lower right corner area of ​​the original image, and then embed the lower right corner of the marked image at this position when embedding the marked image. In this way, for the upper left corner area and the lower right corner area, the upper left vertex and the lower right vertex of the marked image are used as reference points for embedding, which can be more convenient and accurate.

[0112] In the above embodiment, the core position in the target area is set to correspond to the maximum probability value, and the probability distribution of the non-core position decreases around the core position, so that the target position is selected as the embedding position of the marked image according to the probability value corresponding to each position, which is more in line with the actual situation and satisfies random allocation, making the construction of the sample image faster and more accurate.

[0113] In one embodiment, the sample image includes a first sample image and a second sample image; step S208, that is, for each type of labeled image, embedding at least a portion of the labeled image into the original image according to the corresponding embedding position to obtain the corresponding sample image, specifically includes: for each type of labeled image, determining the first labeled image to be completely embedded and the second labeled image to be occluded in the labeled image; completely embedding the first labeled image into the original image according to the corresponding embedding position to obtain the corresponding first sample image; completely embedding the second labeled image into the original image according to the corresponding embedding position, and randomly selecting a target occlusion ratio from a preset occlusion ratio range, and moving a portion of the second labeled image out to the boundary of the original image according to the target occlusion ratio to obtain the corresponding second sample image.

[0114] The first marked image is a marked image selected to be completely embedded in the original image, and the second marked image is a marked image selected to be embedded in the original image as an obstruction. Correspondingly, the first sample image is a sample image corresponding to the first marked image, and the second sample image is a sample image corresponding to the second marked image. It can be understood that the original images here can be the same original image or different original images, and the embodiments of the present application do not limit this.

[0115] In one embodiment, the computer device may pre-set the ratio of the marker images to be fully embedded and to be embedded with occlusion, such as the ratio of the number of first marker images to be fully embedded to the number of second marker images to be embedded with occlusion is 1:3, that is, whether the marker image is occluded is determined with a probability of 0.25. In other words, for each type of marker image, among a certain number of marker images, 25% of the marker images are embedded in the original image with occlusion, and 75% of the marker images are fully embedded in the original image. Of course, the ratio may also be other values, which is not limited in the embodiments of the present application.

[0116] Furthermore, the computer device may determine the first labeled image to be completely embedded and the second labeled image to be embedded with occlusion from all the labeled images. The computer device may completely embed the first labeled image into the original image according to the corresponding embedding position to obtain the corresponding first sample image. When the target area is the upper left corner area, the computer device may embed the upper left vertex of the first labeled image into the embedding position to paste the complete first labeled image into the original image to obtain the corresponding first sample image. When the target area is the lower right corner area, the computer device may embed the lower right vertex of the first labeled image into the embedding position to paste the complete first labeled image into the original image to obtain the corresponding first sample image.

[0117] For the second marked image, the computer device may randomly select a target occlusion ratio from a preset occlusion ratio range, completely embed the second marked image into the original image according to the corresponding embedding position, and then move a portion of the second marked image out to the boundary of the original image according to the selected target occlusion ratio to obtain a corresponding second sample image. The position information of the embedded marked image in the second sample image will be updated according to the moved position.

[0118] The occlusion ratio is the ratio between the height, width or area of ​​the second marked image outside the original image and the original image, such as the height occlusion ratio, width occlusion ratio or area occlusion ratio. The preset occlusion ratio range is a preset range consisting of a series of occlusion ratios, such as the height occlusion ratio range, width occlusion ratio range, or area occlusion ratio range. In other words, the computer device can randomly select a target occlusion ratio from the range, and then move the second marked image upward, downward, left or right toward the boundary of the original image according to the target occlusion ratio, so that the ratio of the height, width or area of ​​the marked image of the second marked image outside the original image to the height, width or area of ​​the second marked image is the target occlusion ratio.

[0119] For example, when the preset occlusion ratio range is a height occlusion ratio range, such as 0.3 to 0.7, the computer device can randomly select a target height occlusion ratio from the corresponding range, such as 0.4, and then when moving the second marker image, the second marker image is translated upward or downward so that a portion of the second marker image is outside the original image, and the height of the portion of the marker image outside the original image is 0.4 times the height of the marker image. When the preset occlusion ratio range is a width occlusion ratio range, such as 0.3 to 0.7, the computer device can randomly select a target width occlusion ratio from the corresponding range, such as 0.4, and then when moving the second marker image, the second marker image is translated left or right so that a portion of the second marker image is outside the original image, and the width of the portion of the marker image outside the original image is 0.4 times the width of the marker image. When the preset occlusion ratio range is an area occlusion ratio range, such as 0.3 to 0.7, the computer device can randomly select a target area occlusion ratio from the corresponding range, such as 0.4, and then when moving the second marked image, move the boundary of the second marked image original image so that a portion of the second marked image is outside the original image, and the area of ​​the portion of the marked image outside the original image is 0.4 times the area of ​​the marked image. It can be understood that the above-mentioned occlusion ratio range and target occlusion ratio are only exemplary values ​​and are not used to limit the present application.

[0120] In one embodiment, for different occlusion ratios in the preset occlusion ratio range, the computer device may set corresponding selection probabilities. The probability values ​​of different occlusion ratios being selected may be the same or different, and this embodiment of the application does not limit this.

[0121] In the above embodiment, when embedding the labeled images into the original images, some labeled images are completely embedded into the original images, and some labeled images are embedded into the original images with occlusion, which can well simulate the situation in real scenes where the labeled images may be edited or compressed, greatly improving the accuracy of training data construction.

[0122] In one embodiment, the detection model training method further includes a step of marking the video to be detected by using the trained detection model, and the step specifically includes:

[0123] S502, obtaining a video to be detected and a trained detection model.

[0124] Specifically, the computer device can obtain the video to be detected and the trained detection model in the video library. In a specific application scenario, the video to be detected is a small video to be detected. Among them, the small video to be detected is a video whose video duration is less than a preset duration or whose video size is less than a preset size.

[0125] S504: extract a preset number of video frames from the video to be detected, and crop each video frame according to the target area to obtain a corresponding target image block.

[0126] Specifically, the computer device can convert the video to be detected into video frames one by one at a preset frequency. For example, the computer device can extract a frame of image as a video frame every second. Then the computer device can filter out a preset number of video frames from multiple video frames, such as filtering out the first frame and the middle frame as the video frames to be detected. Among them, based on practical experience, it can be known that for a small video, there is a high probability that a watermark image (that is, a marked image) will appear in the upper left corner area of ​​the first frame, and there is a high probability that a watermark image will appear in the lower right corner area of ​​the middle frame. Therefore, selecting the first frame and the middle frame as the video frames to be detected can improve the accuracy of the mark detection.

[0127] Then, the computer device can crop the image data of the target area in each video frame to obtain the corresponding target image block, specifically, the upper left corner area and the lower right corner area in each video frame can be cropped to obtain the corresponding target image block.

[0128] S506, input each target image block into the trained detection model respectively, and output the detection result corresponding to each target image block.

[0129] Specifically, the computer device can input each target image block as input data into a trained detection model, process the input data through the model structure and model parameters of the detection model, and output the detection results corresponding to each target image block.

[0130] It can be understood that when the training labels of the detection model during the training process are only the label categories of the label images, the detection results output after processing by the trained detection model are the label categories corresponding to each target image block. When the training labels of the detection model during the training process are the label categories and position information of the label images, the detection results output after processing by the trained detection model are the label categories and position information corresponding to each target image block.

[0131] In one embodiment, step S506, that is, inputting each target image block into the trained detection model respectively, and outputting the detection results corresponding to each target image block specifically includes: inputting each target image block into the trained detection model respectively; processing the input target image block in turn through at least three groups of convolution groups in the trained detection model; wherein the downsampling layer in the last group of convolution groups is a hole convolution with a step size of a preset value, so as to keep the size of the feature map output by the last group of convolution groups as a preset size; performing convolution processing on the feature map output by the middle group of convolution groups to obtain a first feature map to be detected; using the feature map output by the last group of convolution groups as a second feature map to be detected; performing convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected; performing detection processing on the first feature map to be detected, the second feature map to be detected and the third feature map to be detected respectively to obtain the corresponding candidate detection results and the confidence corresponding to the candidate results; from the candidate detection results corresponding to each feature map to be detected, screening out the candidate detection results whose corresponding confidences meet the high confidence conditions as the detection results corresponding to the input target image block.

[0132] Among them, the convolution group (block) is a network structure including multiple convolution layers. Specifically, the trained detection model includes at least three groups of convolution groups, and the computer device inputs the target image block into the trained convolution network, and processes the target image block through each group of convolution groups in the trained convolution network. After each target image passes through a convolution group, the width and height of the corresponding feature map will be reduced by half. The computer device can set the downsampling layer in the last group of convolution groups to use a hole convolution with a step size of a preset value (such as 2), so that the size of the last group of convolution groups is a preset size. Then the computer device can perform convolution processing on the feature map output by the convolution group of the middle group (such as the second group) to obtain a first feature map to be detected, and use the feature map output by the last group of convolution groups as the second feature map to be detected. In addition, the computer device can also perform different convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected, and the sizes of each third feature map to be detected are different.

[0133] Then, the computer device can perform detection on multiple layers of feature maps to be detected, wherein each layer has its own detector (these detectors do not share model parameters), and each detector outputs a candidate detection result (specifically, a regression box and a classification result) for each position of the corresponding feature map to be detected. For each layer of feature maps to be detected, the computer device can use the candidate detection result with the highest confidence among the candidate detection results at each position in the feature map to be detected as the candidate detection result corresponding to the feature map to be detected. Then, the computer device can filter out the candidate detection results whose corresponding confidences meet the high confidence condition from the candidate detection results corresponding to each feature map to be detected as the detection result corresponding to the input target image block. Among them, the confidence that meets the high confidence condition can specifically be the maximum confidence or the top N (N is a positive integer greater than 1) names after sorting the confidences from high to low.

[0134] In a specific embodiment, the computer device may use a detection algorithm based on a neural network. The detection algorithm based on a neural network can be roughly divided into a one-stage detection algorithm and a two-stage detection algorithm. Considering that the one-stage detection algorithm is faster, the detection model in this application may specifically use the SSD detection algorithm in the one-stage detection algorithm. Figure 6 , Figure 6 The schematic diagram of the network structure of a detection network based on the SSD algorithm in one embodiment. The backbone network of the detection network can adopt a ResNet-34 layer network, input a target image block of size 300x300, and then perform detection on the multi-layer feature map to be detected of the network, each layer has its own detector (these detectors do not share parameters), and each detector outputs the regression box and classification result of each position of the feature map to be detected.

[0135] The detection network mentioned in the embodiment of the present application contains one convolution layer and 4 Block layers in sequence, and the width and height of the feature map will be halved after each Block. The feature map size of the target image block after the second Block layer output is 38x38, and the feature map size after the third Block layer is 19x19. In order to make the feature map output by the fourth Block layer larger, the computer device uses a hole convolution with a step size of 2 to implement the downsampling layer of the fourth Block, keeping the size of the output feature map still 19x19, but the feature expression ability is stronger. The 38x38 feature map output by the second Block layer outputs the first feature map to be detected (size is 38x38) after two layers of convolution layers, and the 19x19 feature map corresponding to the fourth Block layer is the second feature map to be detected. After the second feature map to be detected passes through 3 two-layer convolution structures (the second convolution step size is 2), the third feature map to be detected of 10x10, 5x5, and 3x3 is output in sequence. There are a total of 5 feature maps to be detected, namely the first feature map to be detected, the second feature map to be detected and the third feature map to be detected, and each feature map has a corresponding detector (detector header). For each position of the feature map to be detected, a preset number of anchor frames are designed, and the detector will predict the probability that these preset number of anchor frames belong to each marked category and the rectangular bounding box of the marked image. Among them, the probability that each anchor frame belongs to each marked category and the rectangular bounding box of the marked image are the candidate detection results corresponding to each position. The computer device can use the candidate detection result with the highest confidence among the candidate detection results of each position in the 5 feature maps to be detected as the candidate detection result corresponding to the target image block.

[0136] In the above embodiment, by marking and detecting the multiple images to be detected corresponding to the target image block respectively, the candidate detection results with higher confidence levels are screened out from the candidate detection results corresponding to each feature map to be detected as the detection results corresponding to the target image block, thereby ensuring the accuracy of the detection results.

[0137] In one embodiment, the detection model can also be implemented by a RetinaNet network, wherein the RetinaNet network adopts a pyramid network structure, and the pyramid network structure adds a top-down side branch, which amplifies high-level features and adds them bitwise with low-level features, thereby enhancing the expression ability of low-level features. Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the network structure of a RetinaNet network in one embodiment. Figure 7As shown, the network structure of the RetinaNet network is mainly composed of (a) ResNet, (b) FPN and two FCN sub-networks. Among them, ResNet, the full name of Residual Network, is also called residual network. FPN, the full name of Feature Pyramid Network, is also called feature pyramid network. FCN, the full name of Fully Convolutional Networks, is also called full convolutional network. The two fully convolutional networks in the embodiment of the present application can specifically be (c) a classification subnet (class subnet) and (d) a detection box position regression subnet (box subnet), which are used to predict the label category to which the labeled image belongs and the location information of the labeled image. The backbone network (Backbone) of the RetinaNet network can be composed of ResNet+FPN. After the input image is subjected to feature extraction by Backbone, a feature pyramid can be obtained. After obtaining the feature pyramid, continue to refer to Figure 7 , two sub-networks are used for each layer of the feature pyramid, namely the classification sub-network (class subnet) and the detection box position regression sub-network (box subnet), to process and output the final detection results.

[0138] In a specific embodiment, the RetinaNet backbone network uses the ResNet50 network, and the size of the input target image block is 600x600. In practical applications, for image data, there are often foreground objects that are much smaller than the background, and the detector predicts the category probability at each position in the feature map, which will cause the number of background categories to be much greater than the number of foreground categories. In order to solve this problem of great imbalance between the foreground and background, the loss function can be weighted by the Focal Loss (a loss function algorithm) of the RetinaNet network. The advantage of Focal Loss is that it can automatically mine difficult samples and filter out simple samples according to the size of the loss function, so that the network can efficiently learn from difficult samples, that is, it can learn more about the classification of labeled images.

[0139] It can be understood that the detection model can also be implemented using other neural network algorithms. The SSD algorithm and RetinaNet mentioned in the above embodiments are only used for exemplary description and are not used to limit the structure and algorithm of the detection network in the embodiments of the present application.

[0140] S508, fusing the detection results of each target image block to obtain a detection result corresponding to the video to be detected.

[0141] Specifically, the computer device can determine the confidence level of the detection results corresponding to each target image block, and use the detection result of the target image block corresponding to the maximum confidence level as the detection result corresponding to the video to be detected. The confidence level corresponding to the detection result indicates the credibility of the detection result.

[0142] In the above embodiment, the image data of the target area is cropped from the video frame in the video to be detected to obtain the corresponding target image block, and then each target image block is marked and detected through the trained detection model, and the detection results corresponding to each target image block are fused to obtain the detection result corresponding to the video to be detected, so that the detection speed is fast and the detection accuracy is high when performing mark detection on the video to be detected.

[0143] In a specific embodiment, referring to Figure 8 , Figure 8 FIG. 1 is a flow chart of the steps of marking and detecting a video to be detected by using a trained detection model and obtaining a detection result in a specific embodiment. Figure 8 As shown, the computer device obtains a video to be detected, such as a small video on a mobile phone, extracts a video frame from the small video on the mobile phone, and cuts out the target image blocks corresponding to the upper left corner area and the lower right corner area from the extracted video frame. Then the computer device can perform a mark detection on each target image block. In this specific embodiment, the marked image is specifically a watermark image, and thus it can also be called watermark detection. The computer device can fuse the detection results of the watermark detection corresponding to each image block to obtain the detection result corresponding to the video to be detected.

[0144] In practical applications, when the marked image is a watermark image, the detection model obtained by training the detection model training method mentioned in the embodiment of the present application can be used to accurately identify and locate the watermarks of small videos with watermarks of various platforms, with high detection accuracy. Moreover, as various new platforms emerge in an endless stream and the watermarks related to the small videos derived therefrom may also increase, the detection model training method mentioned in the embodiment of the present application can automatically generate new training data and retrain the detection model, so that the new watermarks can be well identified, which greatly reduces the waste of human resources and greatly improves the training efficiency and accuracy.

[0145] In a specific application scenario, the detection result includes the label category to which the labeled image in the video to be detected belongs; the detection model training method also includes the step of filtering a specific video, which specifically includes: obtaining a video filtering instruction; the video filtering instruction includes a first target category; determining the label category corresponding to each video in the video library through the trained detection model; searching the video library for videos to be filtered whose label category is the first target category; in response to the video filtering instruction, pushing videos in the video library except the video to be filtered to the user terminal that initiated the video filtering instruction.

[0146] In a specific application scenario, when a user initiates a video filtering instruction to a computer device through a user terminal, the computer device can extract the first target category carried in the video filtering instruction. The computer device can perform tag detection on each video in the video library through a trained detection model to obtain a detection result corresponding to each video, and the detection result includes a tag category corresponding to each video. Then, the computer device can search for videos to be filtered whose tag category is the first target category from the video library, and push the videos after filtering out the videos to be filtered from the video library to the corresponding user terminal.

[0147] In one embodiment, the marked image is a watermark image, and the computer device can filter videos containing specific watermark images according to the video filtering instruction, so that the user can filter out videos from specific video applications according to his or her likes and dislikes.

[0148] In the above embodiment, the tag category corresponding to each video in the video library can be determined through the trained detection model, and then videos of a certain tag category that the user does not like can be filtered out, so that video services can be provided to the user conveniently and intelligently.

[0149] In a specific application scenario, the detection result includes the label category to which the labeled image in the video to be detected belongs; the detection model training method also includes the step of searching for a specific video, which step specifically includes: obtaining a video search instruction; the video search instruction includes a second target category; determining the label category corresponding to each video in the video library through the trained detection model; searching the video library for a target video whose label category is the second target category; and in response to the video search instruction, pushing the target video to the user terminal that initiated the video instruction.

[0150] In one embodiment, when a user initiates a video search instruction to a computer device through a user terminal, the computer device may extract the second target category carried in the video search instruction. The computer device may perform tag detection on each video in the video library through a trained detection model to obtain detection results corresponding to each video, and the detection results include the tag categories corresponding to each video. Then, the computer device may search for a target video with a tag category of the second target category from the video library, and push the target video to the user terminal that initiated the video instruction.

[0151] In one embodiment, the marked image is a watermark image, and the user can input a watermark category related to the video application so that the computer device can quickly retrieve the corresponding video, so that the user can search for videos from a specific video application according to his or her preferences.

[0152] In the above embodiment, the tag category corresponding to each video in the video library can be determined through the trained detection model, and then the video of a specific tag category that the user likes can be searched, so that video services can be provided to the user conveniently and intelligently.

[0153] In one embodiment, the detection result includes the tag category and location information of the tagged image in the video to be detected; the detection model training method also includes the step of editing the tagged image in the video, which step specifically includes: the computer device can determine the tag category corresponding to each video in the video library and the location information of the included tagged image through the trained detection model, and then locate the tagged image to realize the editing processing of the tagged image.

[0154] Figure 2 FIG. 1 is a flow chart of a detection model training method in one embodiment. It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0155] In one embodiment, Fig. 9As shown, a detection model training device 900 is provided, which can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device includes an acquisition module 901, a selection module 902, an embedding module 903, a determination module 904 and a training module 905, wherein:

[0156] The acquisition module 901 is used to acquire the original image to be processed and more than one type of labeled images.

[0157] The selection module 902 is used to randomly select a target position from the target area of ​​the original image as the embedding position of the marked image for each type of marked image.

[0158] The embedding module 903 is used to embed at least a part of the marked image into the original image according to the corresponding embedding position for each type of marked image to obtain a corresponding sample image.

[0159] The determination module 904 uses the sample image as a training sample and uses the tag category of the tag image embedded in the sample image as a corresponding training tag.

[0160] The training module 905 is used to train the detection model to be trained through training samples and corresponding training labels.

[0161] In one embodiment, the acquisition module 901 is also used to acquire the original image to be processed and more than one type of marking templates; randomly select a target size ratio from a preset size ratio range; and scale each type of marking template according to the target size ratio based on the size of the original image to obtain a corresponding marking image.

[0162] In one embodiment, the target position includes a core position and a non-core position; the selection module 902 is also used to determine the core position in the target area of ​​the original image; obtain the probability value when the core position is used as the embedding position; the probability value corresponding to the core position is the maximum value of the probability values ​​corresponding to each target position in the target area; according to the distance between each non-core position and the core position in the target area, determine the probability value when each non-core position is used as the embedding position; the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position; for each type of marked image, select the corresponding target position as the embedding position of the marked image according to the probability value corresponding to each target position in the target area of ​​the original image.

[0163] In one embodiment, the selection module 902 is also used to obtain a preset number of platform-specific images; each platform-specific image includes a marker image corresponding to the corresponding platform; according to the coordinates of the target vertices of each marker image in the platform-specific image, the average coordinates corresponding to the target vertex are determined; and the average coordinates are used as the core position in the target area of ​​the original image.

[0164] In one embodiment, the target area includes an upper left corner area and a lower right corner area; the selection module 902 is also used to, when the target area is the upper left corner area, for each type of marking template, select the corresponding target position as the embedded position corresponding to the upper left vertex of the marked image according to the probability values ​​corresponding to each target position in the upper left corner area of ​​the original image; when the target area is the lower right corner area, for each type of marking template, select the corresponding target position as the embedded position corresponding to the lower right vertex of the marked image according to the probability values ​​corresponding to each target position in the lower right corner area of ​​the original image.

[0165] In one embodiment, the sample image includes a first sample image and a second sample image; the embedding module 903 is also used to determine, for each type of labeled image, a first labeled image to be completely embedded and a second labeled image to be embedded with occlusion in the labeled image; the first labeled image is completely embedded into the original image according to the corresponding embedding position to obtain the corresponding first sample image; the second labeled image is embedded into the original image according to the corresponding embedding position, and a target occlusion ratio is randomly selected from a preset occlusion ratio range, and a portion of the second labeled image is moved out to the boundary of the original image according to the target occlusion ratio to obtain the corresponding second sample image.

[0166] In one embodiment, the determination module 904 is also used to determine the tag category of the tag image embedded in each sample image, and the position information of the tag image in the original image; the sample image is used as a training sample, and the tag category and corresponding position information of the tag image embedded in the sample image are used together as the training label of the training sample.

[0167] In one embodiment, the training module 905 is also used to crop the sample image according to the target area to obtain the corresponding sample image block; extract features of each sample image block through the detection model to be trained to obtain the corresponding feature map, and perform detection and output prediction results based on the feature map; according to the difference between the prediction result corresponding to the sample image block and the corresponding training label, adjust the model parameters of the detection model until the training stop condition is met.

[0168] In one embodiment, the detection model training device 900 also includes a marker detection module 906, which is used to obtain a video to be detected and a trained detection model; extract a preset number of video frames from the video to be detected, and crop each video frame according to the target area to obtain a corresponding target image block; input each target image block into the trained detection model, and output the detection result corresponding to each target image block; and fuse the detection results of each target image block to obtain a detection result corresponding to the video to be detected.

[0169] In one embodiment, the marker detection module 906 is also used to input each target image block into a trained detection model respectively; the input target image block is processed in turn by at least three convolution groups in the trained detection model; wherein the downsampling layer in the last convolution group is a hole convolution with a step size of a preset value to keep the size of the feature map output by the last convolution group at a preset size; the feature map output by the convolution group of the middle group is convolved to obtain a first feature map to be detected; the feature map output by the last convolution group is used as the second feature map to be detected; the second feature map to be detected is convolved to obtain at least one third feature map to be detected; the first feature map to be detected, the second feature map to be detected and the third feature map to be detected are respectively detected to obtain the corresponding candidate detection results and the confidence corresponding to the candidate results; from the candidate detection results corresponding to each feature map to be detected, the candidate detection results whose corresponding confidences meet the high confidence conditions are screened out as the detection results corresponding to the input target image block.

[0170] refer to Fig.10 In one embodiment, the detection result includes the label category to which the label image in the video to be detected belongs; the detection model training device 900 also includes a video push module 907, which is used to obtain a video filtering instruction; the video filtering instruction includes a first target category; the label category corresponding to each video in the video library is determined by the trained detection model; the video library is searched for videos to be filtered whose label category is the first target category; in response to the video filtering instruction, the videos in the video library except the video to be filtered are pushed to the user terminal that initiates the video filtering instruction.

[0171] In one embodiment, the video push module 907 is also used to obtain a video search instruction; the video search instruction includes a second target category; determine the tag category corresponding to each video in the video library through a trained detection model; search for a target video whose tag category is the second target category from the video library; and in response to the video search instruction, push the target video to the user terminal that initiated the video instruction.

[0172] The above detection model training device randomly embeds each type of labeled image into the original image, and during embedding, simulates the situation in which the labeled image may be edited or compressed in the real scene, with full embedding and partial embedding, so that labeled training data can be automatically generated for training the detection model. Among them, the training label in the training data is the label category to which the embedded labeled image belongs. In this way, there is no need to manually label the training data, and a variety of random strategies are used to simulate the situation where the real labeled image appears in the original image, which greatly reduces the labor cost and labeling efficiency of labeling training data, thereby greatly improving the efficiency of model training.

[0173] For the specific definition of the detection model training device, please refer to the definition of the detection model training method above, which will not be repeated here. Each module in the above-mentioned detection model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0174] Fig.11 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 or the server 120 in FIG. Fig.11 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement the detection model training method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute the detection model training method.

[0175] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0178] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A detection model training method, characterized in that: The method comprises: Obtaining an original image to be processed and labeled images of more than one category; The average position corresponding to the target vertex of the marked image in a preset number of platform-specific images is used as the core position in the target area of ​​the original image, and the other positions of the target position in the target area except the core position are used as non-core positions; the probability value corresponding to the core position is the maximum value among the probability values ​​corresponding to the target positions in the target area, and the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position; For each type of labeled image, according to the probability values ​​corresponding to the target positions in the target area of ​​the original image, the corresponding target positions are selected as the embedding positions of the labeled image; For each type of labeled image, determining a first labeled image to be completely embedded and a second labeled image to be occluded and embedded in the labeled image; Completely embedding the first marked image into the original image according to the corresponding embedding position to obtain a corresponding first sample image; The second marked image is completely embedded into the original image according to the corresponding embedding position, and a target occlusion ratio is randomly selected from a preset occlusion ratio range, and a part of the second marked image is moved out to the boundary of the original image according to the target occlusion ratio to obtain a corresponding second sample image; Using a sample image as a training sample, and using the labeled category of the labeled image embedded in the sample image as a corresponding training label; the sample image includes the first sample image and the second sample image; The detection model to be trained is trained using the training samples and corresponding training labels.

2. The method according to claim 1, characterized in that: The obtaining of the original image to be processed and more than one type of labeled images comprises: Obtaining an original image to be processed and more than one type of labeled templates; Randomly select a target size ratio from a preset size ratio range; According to the size of the original image, each type of marking template is scaled according to the target size ratio to obtain a corresponding marking image.

3. The method according to claim 1, characterized in that After determining the core position and the non-core position of each target position within the target area of ​​the original image, the method further includes: Obtaining a probability value when the core position is used as an embedding position; According to the distances between each non-core position in the target area and the core position, the probability values ​​of taking each non-core position as the embedding position are determined.

4. The method according to claim 1, characterized in that: The step of using the average position corresponding to the target vertices of the marked images in a preset number of platform-specific images as the core position in the target area of ​​the original image includes: Acquire a preset number of platform-specific images; each of the platform-specific images includes a marked image corresponding to the corresponding platform; According to the coordinates of the target vertices of each marked image in the platform-specific image, respectively, determining the average coordinates corresponding to the target vertices; The average coordinates are used as the core position in the target area of ​​the original image.

5. The method according to claim 1, characterized in that The target area includes an upper left corner area and a lower right corner area; for each type of labeled image, selecting a corresponding target position as the embedding position of the labeled image according to the probability values ​​corresponding to each target position in the target area of ​​the original image, includes: When the target area is the upper left corner area, for each type of marked image, according to the probability values ​​corresponding to each target position in the upper left corner area of ​​the original image, the corresponding target position is selected as the embedding position corresponding to the upper left vertex of the marked image; When the target area is the lower right corner area, for each type of marked image, the corresponding target position is selected as the embedding position corresponding to the lower right vertex of the marked image according to the probability value corresponding to each target position in the lower right corner area of ​​the original image.

6. The method according to claim 1, characterized in that The method of using the sample image as a training sample and using the tag category of the tag image embedded in the sample image as a corresponding training label includes: Determining the marker category of the marker image embedded in each of the sample images and the position information of the marker image in the original image; The sample image is used as a training sample, and the tag category and corresponding position information of the tag image embedded in the sample image are used together as training labels of the training sample.

7. The method according to claim 1, characterized in that The step of training the detection model to be trained by using the training samples and the corresponding training labels includes: Cropping the sample image according to the target area to obtain a corresponding sample image block; Extract features from each of the sample image blocks using the detection model to be trained to obtain a corresponding feature map, and perform detection based on the feature map to output a prediction result; According to the difference between the prediction result corresponding to the sample image block and the corresponding training label, the model parameters of the detection model are adjusted until the training is stopped when the training stop condition is met.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Get the video to be detected and the trained detection model; Extracting a preset number of video frames from the video to be detected, and cropping each of the video frames according to the target area to obtain a corresponding target image block; Input each of the target image blocks into the trained detection model, and output the detection results corresponding to each of the target image blocks; The detection results of each target image block are fused to obtain a detection result corresponding to the video to be detected.

9. The method according to claim 8, characterized in that The step of inputting each of the target image blocks into the trained detection model and outputting the detection results corresponding to each of the target image blocks comprises: Inputting each of the target image blocks into the trained detection model respectively; The input target image block is processed in sequence by at least three convolution groups in the trained detection model; wherein the downsampling layer in the last convolution group is a hole convolution with a step size of a preset value, so as to keep the size of the feature map output by the last convolution group at a preset size; Convolution processing is performed on the feature map output by the convolution group of the middle group to obtain a first feature map to be detected; Using the feature map output by the last convolution group as the second feature map to be detected; Performing convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected; Performing detection processing on the first feature map to be detected, the second feature map to be detected, and the third feature map to be detected respectively, to obtain corresponding candidate detection results and confidence levels corresponding to the candidate results; From the candidate detection results corresponding to each feature map to be detected, the candidate detection results whose corresponding confidences meet the high confidence condition are screened out as the detection results corresponding to the input target image block.

10. The method according to claim 8, characterized in that The detection result includes the tag category to which the marked image in the video to be detected belongs; the method further includes: Obtaining a video filtering instruction; the video filtering instruction includes a first target category; Determine the label category corresponding to each video in the video library by using the trained detection model; Searching the video library for videos to be filtered whose marked categories are the first target categories; In response to the video filtering instruction, the videos in the video library except the video to be filtered are pushed to the user terminal that initiates the video filtering instruction.

11. The method according to claim 8, characterized in that The detection result includes the tag category to which the marked image in the video to be detected belongs; the method further includes: Obtaining a video search instruction; the video search instruction includes a second target category; Determine the label category corresponding to each video in the video library by using the trained detection model; Searching the video library for a target video whose marked category is a second target category; In response to the video search instruction, the target video is pushed to a user terminal that initiates the video instruction.

12. A detection model training device, characterized in that: The device comprises: An acquisition module, used for acquiring an original image to be processed and more than one type of labeled images; A selection module is used to use the average position corresponding to the target vertices of the marked images in a preset number of platform-specific images as the core position in the target area of ​​the original image, and use other positions in the target area except the core position as non-core positions; the probability value corresponding to the core position is the maximum value among the probability values ​​corresponding to the target positions in the target area, and the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position; for each type of marked image, select the corresponding target position as the embedding position of the marked image according to the probability value corresponding to each target position in the target area of ​​the original image; The embedding module is used to determine, for each type of labeled image, a first labeled image to be completely embedded and a second labeled image to be embedded with occlusion in the labeled image; completely embed the first labeled image into the original image according to the corresponding embedding position to obtain a corresponding first sample image; completely embed the second labeled image into the original image according to the corresponding embedding position, and randomly select a target occlusion ratio from a preset occlusion ratio range, and move a part of the second labeled image out to the boundary of the original image according to the target occlusion ratio to obtain a corresponding second sample image; A determination module, taking a sample image as a training sample, and taking a label category of a label image embedded in the sample image as a corresponding training label; the sample image includes the first sample image and the second sample image; The training module is used to train the detection model to be trained through the training samples and corresponding training labels.

13. The device according to claim 12, characterized in that The acquisition module is further used for: Obtaining an original image to be processed and more than one type of labeled templates; Randomly select a target size ratio from a preset size ratio range; According to the size of the original image, each type of marking template is scaled according to the target size ratio to obtain a corresponding marking image.

14. The device according to claim 12, characterized in that The acquisition module is further used for: Obtaining a probability value when the core position is used as an embedding position; According to the distances between each non-core position in the target area and the core position, the probability values ​​of taking each non-core position as the embedding position are determined.

15. The device according to claim 12, characterized in that The acquisition module is further used for: Acquire a preset number of platform-specific images; each of the platform-specific images includes a marked image corresponding to the corresponding platform; According to the coordinates of the target vertices of each marked image in the platform-specific image, respectively, determining the average coordinates corresponding to the target vertices; The average coordinates are used as the core position in the target area of ​​the original image.

16. The device according to claim 12, characterized in that The target area includes an upper left corner area and a lower right corner area; the selection module is further used to: When the target area is the upper left corner area, for each type of marked image, according to the probability values ​​corresponding to each target position in the upper left corner area of ​​the original image, the corresponding target position is selected as the embedding position corresponding to the upper left vertex of the marked image; When the target area is the lower right corner area, for each type of marked image, the corresponding target position is selected as the embedding position corresponding to the lower right vertex of the marked image according to the probability value corresponding to each target position in the lower right corner area of ​​the original image.

17. The device according to claim 12, characterized in that The determining module is further used for: Determining the marker category of the marker image embedded in each of the sample images and the position information of the marker image in the original image; The sample image is used as a training sample, and the tag category and corresponding position information of the tag image embedded in the sample image are used together as training labels of the training sample.

18. The device according to claim 12, characterized in that The training module is also used for: Cropping the sample image according to the target area to obtain a corresponding sample image block; Extract features from each of the sample image blocks using the detection model to be trained to obtain a corresponding feature map, and perform detection based on the feature map to output a prediction result; According to the difference between the prediction result corresponding to the sample image block and the corresponding training label, the model parameters of the detection model are adjusted until the training is stopped when the training stop condition is met.

19. The device according to any one of claims 12 to 18, characterized in that The device further comprises a mark detection module, wherein the mark detection module is configured to: Get the video to be detected and the trained detection model; Extracting a preset number of video frames from the video to be detected, and cropping each of the video frames according to the target area to obtain a corresponding target image block; Input each of the target image blocks into the trained detection model, and output the detection results corresponding to each of the target image blocks; The detection results of each target image block are fused to obtain a detection result corresponding to the video to be detected.

20. The device according to claim 19, characterized in that The mark detection module is further used for: Inputting each of the target image blocks into the trained detection model respectively; The input target image block is processed in sequence by at least three convolution groups in the trained detection model; wherein the downsampling layer in the last convolution group is a hole convolution with a step size of a preset value, so as to keep the size of the feature map output by the last convolution group at a preset size; Convolution processing is performed on the feature map output by the convolution group of the middle group to obtain a first feature map to be detected; Using the feature map output by the last convolution group as the second feature map to be detected; Performing convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected; Performing detection processing on the first feature map to be detected, the second feature map to be detected, and the third feature map to be detected respectively, to obtain corresponding candidate detection results and confidence levels corresponding to the candidate results; From the candidate detection results corresponding to each feature map to be detected, the candidate detection results whose corresponding confidences meet the high confidence condition are screened out as the detection results corresponding to the input target image block.

21. The device according to claim 19, characterized in that The detection result includes the tag category to which the marked image in the video to be detected belongs; the device also includes a video pushing module for: Obtaining a video filtering instruction; the video filtering instruction includes a first target category; Determine the label category corresponding to each video in the video library by using the trained detection model; Searching the video library for videos to be filtered whose marked categories are the first target categories; In response to the video filtering instruction, the videos in the video library except the video to be filtered are pushed to the user terminal that initiates the video filtering instruction.

22. The device according to claim 19, characterized in that The detection result includes the tag category to which the marked image in the video to be detected belongs; the device also includes a video pushing module for: Obtaining a video search instruction; the video search instruction includes a second target category; Determine the label category corresponding to each video in the video library by using the trained detection model; Searching the video library for a target video whose marked category is a second target category; In response to the video search instruction, the target video is pushed to a user terminal that initiates the video instruction.

23. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 11.

24. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 11.

Images