Virtual object detection method, device, equipment and computer readable storage medium

Abstract

The application provides a virtual object detection method, device and equipment and a computer readable storage medium. The method comprises: acquiring image samples, selecting a to-be-labeled image sample from the image samples, and obtaining remaining image samples that are not selected; combining style information of the remaining image samples with content information of the to-be-labeled image sample to obtain an incremental sample; acquiring virtual object labeling information of the to-be-labeled image sample, determining the virtual object labeling information as label information of the incremental sample and the to-be-labeled image sample, and training an original detection model; continuously selecting a new to-be-labeled image sample from the remaining image samples to generate a new incremental sample, and training the trained original detection model based on the new incremental sample until a training termination condition is met, thereby obtaining a detection model, wherein the detection model is used for virtual object detection on images. Through the application, the accuracy of virtual object detection can be improved based on artificial intelligence technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology in the field of artificial intelligence, and in particular to a virtual object detection method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of image processing technology, detecting objects in images has been widely used in the Internet field. For example, by detecting objects in images, image-based retrieval and recommendation processing can be achieved. However, when the object in the image is a real object (such as a real person or a real object), the detection model trained by artificial intelligence technology can accurately detect the real object. When the object in the image is a virtual object, the detection accuracy of the detection model trained by artificial intelligence is low due to the small amount of image data set corresponding to the virtual object. SUMMARY

[0003] The embodiments of the present application provide a virtual object detection method, device, equipment and computer readable storage medium, which can improve the accuracy of virtual object detection.

[0004] The technical solutions of the embodiments of the present application are as follows:

[0005] The embodiments of the present application provide a virtual object detection method, comprising:

[0006] Obtain image samples, and select a to-be-labeled image sample from the image samples, thereby obtaining remaining image samples that are not selected, wherein each sample image in the image samples includes a virtual object region;

[0007] Combine the style information of the remaining image samples with the content information of the to-be-labeled image sample to obtain an incremental sample;

[0008] Obtain virtual object labeling information of the to-be-labeled image sample, and determine the virtual object labeling information as label information of the incremental sample and the to-be-labeled image sample, and train an original detection model;

[0009] Continue to select a new to-be-labeled image sample from the remaining image samples to generate a new incremental sample, and train the trained original detection model based on the new incremental sample, until a training termination condition is met, thereby obtaining a detection model, wherein the detection model is used for virtual object detection of images.

[0010] The embodiments of the present application provide a virtual object detection device, comprising:

[0011] The sample selection module is configured to obtain image samples, and select image samples to be labeled from the image samples, thereby obtaining remaining image samples that are not selected, wherein each sample image in the image samples comprises a virtual object region.

[0012] The style conversion module is configured to combine style information of the remaining image samples and content information of the image samples to be labeled, and obtain incremental samples.

[0013] The model training module is configured to obtain virtual object labeling information of the image samples to be labeled, determine the virtual object labeling information as label information of the incremental samples and the image samples to be labeled, and train the original detection model.

[0014] The model iteration module is configured to continue to select new image samples to be labeled from the remaining image samples to generate new incremental samples, and train the trained original detection model based on the new incremental samples, until a training stop condition is met, thereby obtaining a detection model, wherein the detection model is configured to perform virtual object detection on images.

[0015] In the embodiments of the present application, the sample selection module is further configured to divide the image samples into multiple levels based on image difference information, thereby obtaining multiple level image samples, wherein the image difference information is at least one of virtual object quantity information, image quality information, lighting information, virtual object position information, viewing angle information, and virtual style information; and a quantity threshold of sample images selected from each level image sample in the multiple level image samples is combined as the image samples to be labeled.

[0016] In the embodiments of the present application, when the image difference information is the virtual object quantity information, the sample selection module is further configured to obtain the original detection model, perform virtual object detection on each sample image through the original detection model, obtain at least one prediction box probability value corresponding to at least one virtual object prediction box, wherein each virtual object prediction box in the at least one virtual object prediction box is used to label the virtual object region; compare the at least one prediction box probability value with a probability threshold respectively; determine a prediction box quantity corresponding to the at least one virtual object prediction box based on a comparison result; and determine the prediction box quantity as the virtual object quantity information.

[0017] In the embodiments of the present application, the sample selection module is further configured to calculate a probability entropy corresponding to the at least one prediction box probability value; and determine the probability entropy as the virtual object quantity information.

[0018] In the embodiment of the present application, the sample selection module is further configured to: obtain entity object label information corresponding to auxiliary image samples, wherein each auxiliary sample image in the auxiliary image samples comprises an entity object region; train an auxiliary detection model based on the auxiliary image samples and the entity object label information, wherein the auxiliary detection model is configured to perform entity object detection on images; perform virtual object detection on each sample image by using the auxiliary detection model to obtain an initial detection result; combine sample images with at least one virtual object detection box in the initial detection result in the image samples as initial image samples; combine content information of the auxiliary image samples and style information of the initial image samples to obtain transition image samples; and train the auxiliary detection model based on the transition image samples and the entity object label information to obtain the original detection model.

[0019] In the embodiment of the present application, when the image difference information is the image quality information, the sample selection module is further configured to: obtain an image quality score of each sample image in the image samples; and determine the image quality score as the image quality information corresponding to each sample image.

[0020] In the embodiment of the present application, the model training module is further configured to: perform virtual object detection on each sample in the incremental samples and the image samples to be labeled by using the original detection model to obtain at least one prediction probability value corresponding to at least one virtual object box; obtain at least one label box corresponding to each sample from the virtual object label information; determine a virtual object detection loss based on the at least one prediction probability value corresponding to the at least one virtual object box and the at least one label box; and perform back propagation in the original detection model based on the virtual object detection loss to train the original detection model.

[0021] In the embodiment of the present application, the style conversion module is further configured to: perform feature extraction on the image samples to be labeled to obtain content features, and perform feature extraction on the remaining image samples to obtain style features; perform distribution calculation on each channel style feature in the style features to obtain style feature distribution information, and perform distribution calculation on each channel content feature in the content features to obtain content feature distribution information; map the content feature distribution information to the style feature distribution information to obtain to-be-decoded image features; and decode the to-be-decoded image features to obtain the incremental samples.

[0022] In the embodiment of the present application, the feature extraction, the distribution calculation, the mapping and the decoding are implemented through a style model; the detection device further comprises a style model training module configured to initialize model parameters of an original style model; fuse a style training sample and a content training sample through the original style model to obtain a predicted style image; perform back propagation in the original style model based on a style difference between the style training sample and the predicted style image and a content difference between the content training sample and the predicted style image to obtain the style model.

[0023] In the embodiment of the present application, the detection device further comprises a condition judging module configured to obtain a current trained original detection model; calculate a detection rate of the current trained original detection model based on a test data set, wherein the detection rate comprises at least one of an accuracy rate and a recall rate; and determine that the training stop condition is met when the detection rate is greater than a detection rate threshold.

[0024] In the embodiment of the present application, the condition judging module is further configured to obtain a number of labeled sample images in the image sample; calculate a labeling proportion of the number of labeled sample images and a sample number of the image sample; and determine that the training stop condition is met when the labeling proportion is greater than a proportion threshold.

[0025] In the embodiment of the present application, the detection device further comprises an object detection module configured to obtain a virtual object detection request; obtain a to-be-detected image in response to the virtual object detection request; perform virtual object detection on the to-be-detected image through the detection model to detect at least one virtual object region; and perform application processing based on the at least one virtual object region, wherein the application processing is at least one of recommendation processing, retrieval processing and tracking processing.

[0026] An embodiment of the present application provides a virtual object detection device, comprising:

[0027] a memory configured to store executable instructions;

[0028] a processor configured to execute the executable instructions stored in the memory to implement the virtual object detection method provided in the embodiments of the present application.

[0029] An embodiment of the present application provides a computer readable storage medium storing executable instructions for causing a processor to execute the virtual object detection method provided in the embodiments of the present application.

[0030] The embodiments of the present application have at least the following beneficial effects: by selecting the to-be-labeled image samples from the image samples for labeling, generating the incremental samples based on the style information of the remaining image samples that are not selected, and taking the labeled information as the label common to the to-be-labeled image samples and the incremental samples, the original detection model is trained; then, based on the remaining image samples, the above-mentioned selective labeling, sample generation and training process are cyclically executed until the final detection model is obtained; in this way, since the incremental samples contain the style information of the remaining image samples in the training samples used in the detection model acquisition process, when the incremental samples and the selected samples are used to perform model training, and the above-mentioned selective labeling, sample generation and training process are cyclically executed based on the remaining image samples, the model can be effectively trained by comprehensively using the overall information of the image samples under the condition of labeling part of the samples, and further, the learning ability of the model can be improved and the accuracy of virtual object detection can be improved under the condition of limited training samples. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is an optional architecture schematic diagram of a virtual object detection system provided by the embodiments of the present application;

[0032] Figure 2 is a component structure schematic diagram of a server in Figure 1 the embodiments provided by the present application;

[0033] Figure 3 is an optional flow schematic diagram of a virtual object detection method provided by the embodiments of the present application;

[0034] Figure 4 is another optional flow schematic diagram of a virtual object detection method provided by the embodiments of the present application;

[0035] Figure 5 is a flow schematic diagram of an exemplary virtual object detection method provided by the embodiments of the present application;

[0036] Figure 6 is a network structure schematic diagram of an exemplary real person detection model provided by the embodiments of the present application;

[0037] Figure 7 is an exemplary style model training flow schematic diagram provided by the embodiments of the present application;

[0038] Figure 8 is an exemplary automatic labeling human body schematic diagram provided by the embodiments of the present application;

[0039] Figure 9 is an exemplary labeling tool page schematic diagram provided by the embodiments of the present application. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. The described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0041] In the following description, “some embodiments” are related to a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] The relevant data collection and processing in the embodiments of the present application should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0044] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0045] 1) Artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Therefore, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Or, artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0046] In addition, artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning and other major directions. And, with the research and progress of artificial intelligence technology, artificial intelligence technology has carried out research and application in multiple fields; for example, common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, autonomous driving, drones, robots, smart medical and smart customer service, etc.; with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. The application of artificial intelligence involved in the embodiments of the present application in the field of image processing will be described later.

[0047] 2) Machine Learning (ML), is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a branch of artificial intelligence that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills; reorganize existing knowledge structure to continuously improve its performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications span all areas of artificial intelligence. Machine learning usually includes deep learning, artificial neural networks, belief networks, reinforcement learning, transfer learning and inductive learning.

[0048] 3) Virtual objects, various people and objects in the virtual scene that can interact, or movable objects in the virtual scene; the movable object can be a virtual character, a virtual animal, an animation character, etc., such as: a person, an animal, a plant, an oil drum, a wall, a stone, etc. displayed in the virtual scene. In addition, the virtual object can be a virtual image in the virtual scene that represents the user, and the virtual scene can include multiple virtual objects, each virtual object having its own shape and volume in the virtual scene, occupying a part of the virtual scene, corresponding to the virtual object area in the embodiments of the present application.

[0049] It should be noted that, due to the uncertainty of the characteristics of the virtual objects in the ACG (Animation (animation), Comics (comic) and Games (game)) virtual scene, the annotated data set is less, so when the detection of the area where the virtual object is located is implemented, the area where the virtual object is located in the image is usually manually annotated. On the one hand, for the annotation-intensive annotation task, there are 10-20 human bodies that need to determine the annotation box in one picture, and the manual annotation production efficiency is about 10 pictures per person per day, which cannot meet the massive data demand of model identification, and the efficiency is low. On the other hand, due to the differences in light, position difference, angle difference, ACG specific style and other factors, the influence effect of each image sample on the detection task is also different, so when the same category samples are annotated without difference, a large number of samples with little significance for model improvement will be annotated, and the annotation quality is low. On the other hand, the historical accumulation of real person image annotation data is not used in the virtual object detection in the ACG field, so the auxiliary role of other field data and unsupervised data to the new task cannot be played to the greatest extent, and the data utilization efficiency is low.

[0050] Based on this, the embodiment of the application provides a virtual object detection method, device, equipment and computer readable storage medium, which can improve the accuracy of virtual object detection, and also can improve the acquisition efficiency of training samples and the quality of training samples, and can improve the training efficiency of the detection model. The following describes an exemplary application of a virtual object detection device (hereinafter referred to as a detection device) provided by the embodiment of the application. The detection device provided by the embodiment of the application can be implemented as a variety of types of terminals such as a smart phone, a smart watch, a notebook computer, a tablet computer, a desktop computer, a smart television, a set-top box, a smart vehicle-mounted device, a portable music player, a personal digital assistant, a dedicated message device, a portable game device, etc. The detection device can also be implemented as a server. The following describes an exemplary application when the detection device is implemented as a server.

[0051] Referring to Figure 1 , Figure 1 is an optional architecture schematic diagram of a virtual object detection system provided by the embodiment of the application; as Figure 1 indicated, for supporting a virtual object detection application, in the virtual object detection system 100, the server 400 is connected with the terminal 200 (exemplarily showing the terminal 200-1 and the terminal 200-2) through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. In addition, the virtual object detection system 100 further includes a database 500, which is used to provide data support for the server 400 when the server 400 executes a virtual object detection method; and Figure 1As shown in the middle, the database 500 is independent of the server 400, and the database 500 can also be integrated in the server 400, which is not limited in the embodiments of the present application.

[0052] The terminal 200 is configured to present the virtual scene image on the interface of the player as a to-be-detected image and send the to-be-detected image to the server 400 through the network 300. The terminal 200 is also configured to receive the processing information (for example, the recommendation information received by the terminal 200-1 and the search result received by the terminal 200-2) sent by the server 400 for the to-be-detected image through the network 300 and display the processing information on the graphical interface.

[0053] The server 400 is configured to obtain image samples and select a to-be-labeled image sample from the image samples, thereby obtaining remaining image samples that are not selected, wherein each sample image in the image samples includes a virtual object region; obtain an incremental sample by combining style information of the remaining image samples and content information of the to-be-labeled image sample; obtain virtual object labeling information of the to-be-labeled image sample and determine the virtual object labeling information as label information of the incremental sample and the to-be-labeled image sample, train an original detection model; continue to select a new to-be-labeled image sample from the remaining image samples to generate a new incremental sample, and train the trained original detection model based on the new incremental sample, until a training stop condition is met, thereby obtaining a detection model, wherein the detection model is used for virtual object detection on images. The server 400 is also configured to receive the to-be-detected image sent by the terminal 200 through the network 300, perform virtual object detection on the to-be-detected image through the detection model, determine processing information (for example, the recommendation information sent to the terminal 200-1 and the search result sent to the terminal 200-2) based on at least one virtual object region detected, and send the processing information to the terminal 200 through the network 300.

[0054] In some embodiments, the server 400 can be a physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 200 can be a smart phone, a smart watch, a notebook computer, a tablet computer, a desktop computer, a smart television, a set-top box, a smart vehicle device, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device, and the like, but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0055] Referring toFigure 2 , Figure 2 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the composition structure of a server. Figure 2 The server 400 shown includes at least one processor 410, memory 450, at least one network interface 420, and a user interface 430. The various components in server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0056] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] In some embodiments, the user interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touchscreen display, camera, other input buttons and controls.

[0058] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0059] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0060] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0061] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0062] The network communication module 452 is configured to reach other computer devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), and the like;

[0063] The presentation module 453 is configured to enable presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, and the like) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0064] The input processing module 454 is configured to detect and interpret one or more user inputs or interactions from one or more input devices 432.

[0065] In some embodiments, the virtual object detection apparatus (hereinafter referred to as the detection apparatus) provided by the embodiments of the present application can be implemented in a software manner, Figure 2 The virtual object detection apparatus 455 stored in the memory 450 is shown, which can be in the form of software such as programs and plug-ins, including the following software modules: a sample selection module 4551, a style conversion module 4552, a model training module 4553, a model iteration module 4554, a style model training module 4555, a condition judgment module 4556, and an object detection module 4557. These modules are logical, and thus can be combined or further split according to the functions implemented. The functions of each module will be described below.

[0066] In some embodiments, the detection device provided by the embodiments of the present application can be implemented in a hardware manner. For example, the detection device provided by the embodiments of the present application can be a hardware decoding processor programmed to execute the virtual object detection method provided by the embodiments of the present application. For example, the hardware decoding processor can be implemented by using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.

[0067] The virtual object detection method provided by the embodiments of the present application will be described below in combination with an exemplary application and implementation of a server provided by the embodiments of the present application.

[0068] Referring to Figure 3 , Figure 3 is an optional flowchart of the virtual object detection method provided by the embodiments of the present application, which will be described in combination with the steps shown in Figure 3 .

[0069] S301, an image sample is obtained, and a to-be-labeled image sample is selected from the image sample, thereby obtaining a remaining image sample that is not selected.

[0070] In the embodiments of the present application, when a detection device trains a model for performing virtual object detection, an image for training is obtained, and thus an image sample is obtained. Here, the image sample is a set of original sample images and does not include label information. Therefore, the detection device needs to label the image sample. When labeling, the detection device selects part of the sample images from the image sample for labeling. The selected part of the sample images are the to-be-labeled image sample. In addition, the selection strategy of the to-be-labeled image sample can be a random selection strategy or other selection strategies, such as a selection strategy based on a confidence threshold of a model, a selection strategy based on an arbitrary combination of image features, etc. The embodiments of the present application do not make a specific limitation in this regard.

[0071] It should be noted that after the detection device selects the image sample to be labeled from the image samples, the remaining sample images, i.e., the remaining image samples that are not selected; that is, the remaining image samples are sample images other than the image sample to be labeled in the image samples; therefore, the image samples include the image sample to be labeled and the remaining image samples. In addition, each sample image in the image samples includes a virtual object region, where the virtual object region is the region of the virtual object in the sample image. In addition, virtual object detection is a process of detecting the region of the virtual object, such as a process of detecting the region of a virtual character; for example, a process of detecting the region of a virtual prop.

[0072] S302, in combination with the style information of the remaining image samples and the content information of the image sample to be labeled, obtaining an incremental sample.

[0073] In the embodiments of the present application, after the detection device obtains the image sample to be labeled, since the image sample to be labeled is used for training the model after labeling, and the remaining image samples have no corresponding label information and cannot participate in the training of the model; therefore, in order to enable the remaining image samples to also participate in the model training, the detection device borrows the style information of the remaining image samples, combines the style information of the remaining image samples with the content information of the image sample to be labeled, to generate a new sample image, i.e., an incremental sample.

[0074] It should be noted that the incremental sample has the content information of the image sample to be labeled, and has the style information of the remaining image samples.

[0075] It can be understood that since the image sample to be labeled has corresponding true value labeling data, based on the content information of the image sample to be labeled, and in combination with the style information of the remaining image samples, the incremental sample is generated, without introducing labeling noise, which can improve the recall effect of the model under accurate labeling information, and can speed up the labeling progress and reduce the labeling amount. That is, by combining automatic labeling (generation of incremental samples) and manual labeling, the sample space is expanded in terms of semantics and quantity, thereby improving the training efficiency and detection effect of the model.

[0076] S303, obtaining virtual object labeling information of the image sample to be labeled, and determining the virtual object labeling information as label information of the incremental sample and the image sample to be labeled, and training the original detection model.

[0077] In the embodiments of the present application, after the detection device obtains the image sample to be labeled, the region of the virtual object on the sample image in the image sample to be labeled is labeled, and the corresponding virtual object labeling information of the image sample to be labeled is obtained. Here, the detection device obtains the virtual object labeling information of the image sample to be labeled, and the execution sequence is not prior to or subsequent to S302.

[0078] It should be noted that, since the incremental sample has the same content information as the to-be-labeled image sample, that is, has the same virtual object region, therefore, the label information corresponding to the virtual object labeling information is also the virtual object labeling information; thereby, the detection device can perform model training based on the training data (the to-be-labeled image sample and the virtual object labeling information, and the incremental sample and the virtual object labeling information); wherein, the performed model training is for the original detection model, which refers to the model to be trained for performing virtual object detection.

[0079] S304, continue to select a new to-be-labeled image sample from the remaining image samples to generate a new incremental sample, and train the trained original detection model based on the new incremental sample, until a training stop condition is met, and obtain the detection model.

[0080] In the embodiments of the present application, the processes described in S301 to S303 complete one training for the original detection model. Here, when the detection device completes this training for the original detection model, it determines whether the training stop condition is met. If it is met, the trained original detection model is the final detection model. If it is not met, the detection device continues to train the trained original detection model using processes similar to those described in S301 to S303, until it is determined that the training stop condition is met, and stops training and determines the current trained original detection model as the detection model; that is, the detection device first selects a new to-be-labeled image sample from the remaining image samples, thereby obtaining a new remaining image sample from the remaining image samples excluding the new to-be-labeled image sample (here, the remaining image samples include the new to-be-labeled image sample and the new remaining image sample), then obtains a new incremental sample by combining the style information of the new remaining image sample and the content information of the new to-be-labeled image sample, finally, labels the new to-be-labeled image sample to obtain new virtual object labeling information, and uses the new virtual object labeling information as the label information of the new to-be-labeled image sample and the new incremental sample to train the trained original detection model again; here, the detection device judges the training stop condition after each training, and if it does not meet the condition, it continues the next training, and if it meets the condition, it stops training and obtains the final detection model.

[0081] It should be noted that the detection model is obtained after at least one training of the original detection model, and the detection model is used for virtual object detection on images. In addition, the training stop condition can be model convergence, can also be reaching a training number threshold, can also be that the model meets a detection condition, and can also be the labeling situation of a sample image in an image sample, etc., which is not limited in the embodiments of the present application.

[0082] It can be understood that, by selecting the to-be-labeled image samples from the image samples for labeling, generating the incremental samples based on the style information of the remaining image samples that are not selected, and taking the labeled information as the label common to the to-be-labeled image samples and the incremental samples, the original detection model is trained; then, based on the remaining image samples, the above-mentioned selective labeling, sample generation, and training are cyclically performed until the final detection model is obtained; in this way, since the incremental samples contain the style information of the remaining image samples in the training samples used in the detection model acquisition process, when the model training is performed using the incremental samples and the selected samples, and the above-mentioned selective labeling, sample generation, and training are cyclically performed based on the remaining image samples, the overall information of the image samples can be comprehensively used to effectively train the model in the case of partial labeled samples, and then the learning ability of the model can be improved and the accuracy of virtual object detection can be improved in the case of limited training samples. In addition, through the acquisition of the incremental samples, the number of samples is also increased, and the generalization of the obtained detection model is further improved. In addition, through the selective labeling of the images in the image samples each time the training is performed, a gradual training process is realized, targeted training is achieved, and the training efficiency of the detection model can be improved. In addition, the sample generation based on the style sample expansion is performed once for each newly added labeled sample before each training, so that the model still has high robustness in the case of limited labeled data.

[0083] In the embodiments of the present application, the detection device selects the to-be-labeled image samples from the image samples in S301, which can be implemented through S3011 and S3012, and each step will be described below.

[0084] S3011, based on the image difference information, the image samples are divided into multiple levels to obtain multiple level image samples.

[0085] In the embodiments of the present application, the detection device can randomly select the to-be-labeled image samples from the image samples, and can also selectively select the to-be-labeled image samples from the image samples, which is not limited in the embodiments of the present application. Here, when the detection device selectively selects the to-be-labeled image samples from the image samples, first, the sample images in the image samples are divided into multiple level image samples based on the image difference information. Here, the detection device can sort the sample images in the image samples based on the image difference information, and then divide the sorted image samples into multiple level image samples; or can cluster or classify the image samples based on the image difference information to obtain multiple level image samples; and the like, which is not limited in the embodiments of the present application.

[0086] It should be noted that the image difference information is at least one of virtual object quantity information, image quality information, illumination information, virtual object position information, view angle information and virtual style information. The virtual object quantity information refers to information determined based on the quantity of regions in which virtual objects are included in the sample image; the image quality information refers to information determined based on quality attributes such as definition and sharpness of the sample image; the illumination information refers to information determined based on illumination attributes of the sample image; the virtual object position information refers to information determined based on the positions of regions in which virtual objects are included in the sample image; the view angle information refers to a view angle at which information is presented in the sample image; and the virtual style information refers to style information of the sample image. Here, the image sample includes a plurality of hierarchical image samples.

[0087] S3012, combine the quantity threshold number of sample images selected from each hierarchical image sample in the plurality of hierarchical image samples into the to-be-labeled image sample of the image sample.

[0088] In the embodiment of the present application, the detection device selects the quantity threshold number of sample images from each hierarchical image sample in the plurality of hierarchical image samples, and thus obtains a plurality of quantity threshold number of sample images corresponding to the plurality of hierarchical image samples; here, the detection device combines the plurality of quantity threshold number of sample images, and thus obtains the to-be-labeled image sample of the image sample.

[0089] It should be noted that when the detection device selects the quantity threshold number of sample images from each hierarchical image sample, if the sample images in each hierarchical image sample have an order in the image difference information, the quantity threshold number of sample images can be selected from high to low according to the image difference information, or the quantity threshold number of sample images can be randomly selected, and the embodiment of the present application does not make a specific limitation thereon; and if the sample images in each hierarchical image sample do not have an order in the image difference information, the quantity threshold number of sample images can be randomly selected.

[0090] It can be understood that by grading the image samples and selecting sample images from each grade to combine into the image samples to be labeled, the picture samples in the image samples are treated differently; the image samples to be labeled that the original detection model most needs to learn can be selected; therefore, labeling the image samples to be labeled can ensure that the labeling result maximizes the detection capability of the detection model. On the other hand, by selecting the image samples to be labeled that have labeling value, the remaining image samples are style images that are beneficial to the original detection model most needing to learn; therefore, when generating the incremental samples based on the style information of the remaining image samples to train the original detection model, the detection capability of the original detection model can be improved. In addition, by means of active learning: each round divides the unlabeled samples into grades, and uniformly extracts samples covering all grades for labeling to update the model, which combines the characteristics that different samples have different effects on the task, effectively avoids the high mislabeling and missing labeling rate of manual work on pictures, reduces the labeling cost while improving the labeling quality.

[0091] In the embodiment of the present application, when the image difference information is image quality information, before the detection device grades the image samples based on the image difference information in S3011, the virtual object detection method further includes S3013 and S3014, which will be described below.

[0092] S3013, obtaining an image quality score of each sample image in the image samples.

[0093] In the embodiment of the present application, the detection device scores each sample image in the image samples based on the image quality, and thus obtains the image quality score of each sample image. Here, the image quality score can be positively correlated with the image quality of the sample image.

[0094] S3014, determining the image quality score as the image quality information corresponding to each sample image.

[0095] In the embodiment of the present application, the image quality score is the image quality information, so that the detection device grades the image samples based on the image quality score, and the difference between the multiple grade image samples obtained in image quality is large, while the difference between the sample images in one grade image sample in image quality is small.

[0096] In the embodiment of the present application, when the image difference information is virtual object quantity information, before the detection device grades the image samples based on the image difference information in S3011, the virtual object detection method further includes S3015 and S3019, which will be described below.

[0097] S3015, obtaining an original detection model.

[0098] It should be noted that the detection device can obtain the original detection model by obtaining a pre-trained model, for example, obtaining a model for performing entity object detection processing and performing parameter initialization; can also obtain the original detection model by directly initializing the parameters of the selected model structure; and the like, which is not limited in the embodiments of the present application.

[0099] S3016, performing virtual object detection on each sample image through the original detection model to obtain at least one prediction box probability value corresponding to at least one virtual object prediction box.

[0100] In the embodiments of the present application, the detection device performs virtual object detection on each sample image in the sample image through the original detection model, and the obtained result is at least one virtual object prediction box on each sample image and at least one prediction box probability value corresponding to the at least one virtual object prediction box.

[0101] It should be noted that each virtual object prediction box in the at least one virtual object prediction box is used to determine the region of a virtual object in the sample image, that is, each virtual object prediction box is used to label the virtual object region. Each prediction box probability value in the at least one prediction box probability value is used to represent the possibility that the region in the corresponding virtual object prediction box is the region of the virtual object.

[0102] S3017, comparing the at least one prediction box probability value with the probability threshold value respectively.

[0103] In the embodiments of the present application, the detection device can obtain the probability threshold value, which is used to determine whether the region in each virtual object prediction box is the region of the virtual object; thus, after comparing the size relationship between each prediction box probability value in the at least one prediction box probability value and the probability threshold value, the detection device can determine whether the virtual object prediction box corresponding to the prediction box probability value is the region of the virtual object.

[0104] S3018, determining the number of prediction boxes corresponding to the at least one virtual object prediction box based on the comparison result.

[0105] It should be noted that the comparison result obtained by the detection device is the size relationship between each of the at least one prediction box probability value and the probability threshold value; if one prediction box probability value is greater than the probability threshold value, it is determined that the virtual object prediction box corresponding to the prediction box probability value is the region where the virtual object is located, and the number of prediction boxes is recorded as 1; and if one prediction box probability value is less than or equal to the probability threshold value, it is determined that the virtual object prediction box corresponding to the prediction box probability value is not the region where the virtual object is located, and does not participate in the counting of the number of prediction boxes; when the detection device completes the processing of the at least one virtual object prediction box, the number of prediction boxes is also counted. That is, the number of prediction boxes refers to the number of virtual object prediction boxes in which the prediction box probability value is greater than the probability threshold value in the at least one virtual object prediction box.

[0106] S3019, determining the number of prediction boxes as the virtual object number information.

[0107] In the embodiment of the present application, the number of prediction boxes is the virtual object number information, so that the detection device classifies the image samples based on the number of prediction boxes, and the difference between the multiple level image samples obtained in the number of prediction boxes is large, and the difference between the sample images in one level image sample in the number of prediction boxes is small.

[0108] In the embodiment of the present application, after the detection device obtains at least one prediction box probability value corresponding to at least one virtual object prediction box in S3016, the virtual object detection method further includes S30110 and S30111, which will be described below.

[0109] S30110, calculating the probability entropy corresponding to the at least one prediction box probability value.

[0110] In the embodiment of the present application, the detection device calculates the entropy of each sample image based on the at least one prediction box probability value, and thus obtains the probability entropy.

[0111] S30111, determining the probability entropy as the virtual object number information.

[0112] It should be noted that the probability entropy represents the number information of the region where the virtual object is located in each sample image, so the detection device determines the probability entropy as the virtual object number information; thus, the detection device classifies the image samples based on the probability entropy, and the difference between the multiple level image samples obtained in the probability entropy is large, and the difference between the sample images in one level image sample in the probability entropy is small.

[0113] In the embodiment of the present application, S3015 can be implemented by S30151 to S30156; that is, the detection device obtains the original detection model, including S30151 to S30156, which will be described below.

[0114] S30151, obtain entity object annotation information corresponding to the auxiliary image sample.

[0115] It should be noted that when the detection device obtains the original detection model by obtaining the pre-training model, the sample used to train the model for entity object detection is obtained first, and the auxiliary image sample is also obtained, wherein each auxiliary sample image in the auxiliary image sample includes an entity object region, such as a region where a real person is located, a region where a real object is located, and the like. In addition, the auxiliary image sample has corresponding label information, i.e., entity object annotation information, which refers to the region where the entity object in the auxiliary image sample is located.

[0116] S30152, train an auxiliary detection model based on the auxiliary image sample and the entity object annotation information.

[0117] In the embodiment of the present application, the detection device uses the auxiliary image sample and the entity object annotation information for supervised training, and thus obtains an auxiliary detection model for entity object detection of an image. Here, the process of training the auxiliary detection model can include: the detection device performs entity object detection on each auxiliary sample image in the auxiliary image sample by using the original auxiliary detection model, and obtains at least one entity object prediction probability value corresponding to at least one entity object prediction box; obtains at least one entity object detection box corresponding to each auxiliary sample image from the entity object annotation information; determines an entity object detection loss based on the at least one entity object prediction probability value corresponding to the at least one entity object prediction box and the at least one entity object detection box; and performs back propagation in the original auxiliary detection model based on the entity object detection loss, and obtains the auxiliary detection model.

[0118] It should be noted that entity object detection refers to processing of detecting a region where an entity object is located, such as processing of detecting a region where a real person is located, and processing of detecting a region where a real object is located. The original auxiliary detection model is a model to be trained for performing entity object detection.

[0119] S30153, perform virtual object detection on each sample image by using the auxiliary detection model, and obtain an initial detection result.

[0120] In the embodiment of the present application, the detection device inputs each sample image in the image sample to the auxiliary detection model for virtual object detection processing, and the obtained result is an initial detection result, which refers to a situation of a region where a virtual object in each sample image is estimated to be located.

[0121] S30154, combine the sample images with the initial detection result of at least one virtual object bounding box in the image sample into an initial image sample.

[0122] It should be noted that, since the auxiliary detection model is used to perform entity object detection, when the auxiliary detection model is used to perform virtual object detection on each sample image, only the region where the virtual object similar to the entity object can be detected; that is, generally, the sample image similar to the auxiliary sample image can be detected by the auxiliary detection model to obtain the virtual object region, but the sample image greatly different from the auxiliary sample image is difficult to detect the virtual object region. Therefore, the initial detection result may be at least one virtual object bounding box, or may be information that no virtual object bounding box is detected.

[0123] In the embodiment of the present application, the detection device combines the sample images with the initial detection result of at least one virtual object bounding box in the image sample, and obtains the initial image sample; the initial image sample is used to initialize the auxiliary detection model to obtain the pre-training model, that is, the original detection model.

[0124] S30155, combine the content information of the auxiliary image sample and the style information of the initial image sample to obtain a transition image sample.

[0125] It should be noted that the detection device determines the sample image similar to the auxiliary sample image (the initial image sample) determined by the auxiliary detection model from the image sample without label information as a bridge for virtual object detection; that is, the detection device starts training the model for virtual object detection from the easy sample image (the initial image sample).

[0126] In the embodiment of the present application, the detection device combines the style information of the initial image sample with the content information of the auxiliary image sample, and the generated image is a transition image sample; the transition image sample is used to train the auxiliary detection model combined with the label information of the auxiliary image sample. Here, the transition image sample has the content information of the auxiliary image sample, and has the style information of the initial image sample.

[0127] S30156, train the auxiliary detection model based on the transition image sample and the entity object annotation information to obtain the original detection model.

[0128] It should be noted that, since the transition image sample has the same content information as the auxiliary image sample, that is, has the same object region, therefore, the label information corresponding to the transition image sample is also the entity object annotation information. Here, the process of training the auxiliary detection model based on the transition image sample and the entity object annotation information by the detection device is similar to the process of training the detection model, and the embodiment of the present application will not be described here.

[0129] It can be understood that the virtual style transition image sample with entity object annotation information is generated by the auxiliary image sample to train an initial detection model for virtual object detection, and the effective utilization of the entity object annotation information corresponding to the auxiliary image sample and the auxiliary detection model in the field of entity object detection is realized, and the training efficiency of the detection model is improved.

[0130] In the embodiment of the present application, the detection device determines the virtual object annotation information as the label information of the incremental sample and the image sample to be labeled in S303, trains the original detection model, including S3031-S3034, which will be described below.

[0131] S3031, by the original detection model, each sample in the incremental sample and the image sample to be labeled is detected for virtual object, and at least one prediction probability value corresponding to at least one virtual object frame is obtained.

[0132] In the embodiment of the present application, the detection device takes each image in the incremental sample and the image sample to be labeled as a sample to be detected for virtual object, and uses the original detection model to perform virtual object detection processing on the sample; here, for each sample, at least one prediction probability value corresponding to at least one virtual object frame can be detected.

[0133] It should be noted that each virtual object frame in the at least one virtual object frame is used to determine the region of a virtual object in the sample; and each prediction probability value in the at least one prediction probability value is used to represent the possibility that the region in the corresponding virtual object frame is the region of the virtual object.

[0134] S3032, from the virtual object annotation information, at least one annotation frame corresponding to each sample is obtained.

[0135] In the embodiment of the present application, the detection device selects the label information corresponding to the sample to be detected for virtual object from the virtual object annotation information, and obtains at least one annotation frame corresponding to the sample.

[0136] S3033, based on at least one prediction probability value corresponding to at least one virtual object frame, and at least one annotation frame, the virtual object detection loss is determined.

[0137] In the embodiment of the present application, the detection device compares the at least one predicted probability value with the probability threshold value respectively to determine the detection frame result corresponding to the at least one virtual object frame; and determines the difference between the detection frame result and the at least one labeled frame as the loss for training the original detection model, that is, the virtual object detection loss. Here, the detection device can take the virtual object frame corresponding to the predicted probability value greater than the probability threshold value as one of the detection frames in the detection frame result, and discard the virtual object frame corresponding to the predicted probability value less than or equal to the probability threshold value; so that the detection frame in the detection frame result is the virtual object frame with a larger predicted probability value in the at least one virtual object frame.

[0138] S3034, based on the virtual object detection loss, performing back propagation in the original detection model to train the original detection model.

[0139] It should be noted that after the detection device obtains the virtual object detection loss, the detection device performs back propagation in the original detection model based on the virtual object detection loss to adjust the parameters of the original detection model and realize training of the original detection model. Here, the back propagation is described as follows: the sample is input to the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs at least one predicted probability value corresponding to at least one virtual object frame, which is the forward propagation process of the neural network model. Since there is a difference between the output result of the neural network model and the actual result, the difference between the output result and the actual result is calculated, and the difference is back propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back propagation, the value of the model parameter is adjusted according to the error; the above process is iterated until the loss function converges.

[0140] In the embodiment of the present application, S302 can be implemented by S3021 to S3024; that is, the detection device obtains the incremental sample by combining the style information of the remaining image sample and the content information of the image sample to be labeled, including S3021 to S3024, which are described below.

[0141] S3021, performing feature extraction on the image sample to be labeled to obtain content features, and performing feature extraction on the remaining image sample to obtain style features.

[0142] In the embodiment of the present application, the detection device performs feature extraction on the image sample to be labeled and the remaining image sample respectively, and thus obtains the content features of the image sample to be labeled and the style features of the remaining image sample.

[0143] S3022, performing distribution calculation on each channel style feature in the style features to obtain style feature distribution information, and performing distribution calculation on each channel content feature in the content features to obtain content feature distribution information.

[0144] In the embodiments of the present application, the detection device performs distribution calculation on the style features and the content features respectively in units of channels, such as mean value and variance calculation, so as to obtain the style feature distribution information corresponding to the style features composed of the distribution calculation results of each channel in the style features, and the content feature distribution information corresponding to the content features composed of the distribution calculation results of each channel in the content features.

[0145] It should be noted that each channel style feature is a feature of each channel in the style features, and each channel content feature is a feature of each channel in the content features.

[0146] S3023, mapping the content feature distribution information to the style feature distribution information to obtain image features to be decoded.

[0147] In the embodiments of the present application, the detection device maps the content feature distribution information to the style feature distribution information, so that the content feature distribution information is matched with the style feature distribution information, so that the image sample to be labeled has the style features. Here, the feature obtained by the detection device after mapping is the feature of the generated image to be decoded, which is called image features to be decoded.

[0148] S3024, decoding the image features to be decoded to obtain the incremental sample.

[0149] It should be noted that the image obtained by the detection device by decoding the image features to be decoded is the incremental sample.

[0150] In the embodiments of the present application, the feature extraction process in S3021, the distribution calculation process in S3022, the mapping process in S3023 and the decoding process in S3024 can be implemented by a style model; at this time, S3021 further includes the processing steps of training the style model described in S3025 to S3027; that is, before the detection device extracts the features of the image sample to be labeled, the virtual object detection method further includes S3025 to S3027, which will be described below.

[0151] S3025, initializing the model parameters of the original style model.

[0152] It should be noted that the original style model is a model to be trained for fusing the style information of one image and the content information of another image to generate an image.

[0153] S3026, fusing the style training sample and the content training sample by the original style model to obtain a predicted style image.

[0154] In the embodiment of the present application, the detection device inputs the style training sample and the content training sample into the original style model for fusion to generate a new image, that is, a predicted style image.

[0155] S3027, based on the style difference between the style training sample and the predicted style image, and the content difference between the content training sample and the predicted style image, the original style model is back propagated to obtain a style model.

[0156] It should be noted that the predicted style image is the result of the original style model performing style transfer, and the detection device obtains the basis for back propagating the original style model by obtaining the style difference between the style training sample and the predicted style image, and the content difference between the content training sample and the predicted style image; thus, the detection device back propagates in the original style model according to the style difference and the content difference to train the original style model, thereby obtaining a style model. Here, the style model is used to fuse the style information of an image and the content information of another image to generate an image.

[0157] In the embodiment of the present application, when the detection device determines that the training stop condition is met in S304, before obtaining the detection model, the virtual object detection method further includes S305 to S307, which will be described below.

[0158] S305, obtaining a current trained original detection model.

[0159] It should be noted that the current trained original detection model refers to the model obtained after the original detection model has completed the current training, which is the original detection model after at least one training.

[0160] S306, calculating the detection rate of the current trained original detection model based on a test data set.

[0161] It should be noted that the detection device can obtain a data set for estimating the training effect, that is, a test data set; the test data set is the same as the image sample, and the corresponding image includes the region where the virtual object is located, and the test data set corresponds to label information for the region where the virtual object is located. In addition, the detection rate includes at least one of the accuracy and the recall rate, and can also include F1 value and other indicators for measuring the accuracy of the model performing virtual object detection.

[0162] S307, when the detection rate is greater than a detection rate threshold, it is determined that the training stop condition is met.

[0163] In the embodiment of the present application, the detection device can obtain a detection rate threshold, which is used to determine whether the current trained original detection model meets the training expectation. Here, when the detection rate is greater than the detection rate threshold, it is determined that the current trained original detection model meets the training expectation, thereby determining that the training stop condition is met; and when the detection rate is less than or equal to the detection rate threshold, it is determined that the current trained original detection model does not meet the training expectation, thereby continuing to train the current trained original detection model.

[0164] In the embodiment of the present application, when the detection device in S304 determines that the training stop condition is met, before obtaining the detection model, the virtual object detection method further includes S308 to S310, which will be described below.

[0165] S308, the number of labeled sample images in the image sample is obtained.

[0166] It should be noted that the number of sample images refers to the number of sample images that have been labeled in the image sample.

[0167] S309, the labeling proportion of the number of labeled sample images and the number of sample images in the image sample is calculated.

[0168] In the embodiment of the present application, after the detection device obtains the number of labeled sample images, the number of labeled sample images is taken as the numerator, and the number of sample images of the sample images in the image sample is taken as the denominator, and the ratio is calculated, that is, the labeling proportion is obtained.

[0169] S310, when the labeling proportion is greater than the proportion threshold, it is determined that the training stop condition is met.

[0170] It should be noted that the detection device can obtain a proportion threshold, which is used to determine whether the training stop condition is met from the labeling situation of the image sample. Here, when the detection device determines that the labeling proportion is greater than the proportion threshold, it is determined that the training stop condition is met; and when the detection device determines that the labeling proportion is less than or equal to the proportion threshold, it is determined that the training stop condition is not met, thereby continuing to train the current trained original detection model.

[0171] Referring to Figure 4 , Figure 4 is another optional flowchart of the virtual object detection method provided by the embodiment of the present application; as shown in Figure 4 In the embodiment of the present application, S304 further includes S311 to S314; that is, after the detection device obtains the detection model, the virtual object detection method further includes S311 to S314, which will be described below. The virtual object detection method is executed by the detection device, which can be a server.

[0172] S311. Obtain virtual object detection request.

[0173] In this embodiment of the application, when the terminal (e.g., Figure 1 When the terminal (200) receives a retrieval operation applied to the retrieval control (e.g., when an image of a virtual scene is input into the image retrieval control for image retrieval), it sends a virtual object detection request to the detection device in response to the retrieval operation, and the detection device obtains the virtual object detection request; or, when the terminal triggers virtual object detection processing for the currently playing virtual scene video (e.g., when the playback progress of the currently playing virtual scene video meets the trigger condition, such as when it is about to finish playing), the terminal sends a virtual object detection request to the detection device for the video of the virtual scene, and the detection device obtains the virtual object detection request; or, when the terminal receives a tracking operation applied to the tracking control (e.g., when determining the motion trajectory of a virtual object in the currently playing virtual scene video), the terminal sends a virtual object detection request to the detection device for the video of the virtual scene, and the detection device obtains the virtual object detection request. Here, the virtual object detection request is used to request the execution of virtual object detection processing.

[0174] S312. In response to the virtual object detection request, acquire the image to be detected.

[0175] It should be noted that after the detection device receives a virtual object detection request, it responds to the virtual object detection request by triggering the virtual object detection process. First, it obtains the object to be detected as indicated by the virtual object detection request, such as an image of the virtual scene input by the terminal side, or a video frame image in the video of the virtual scene, thus obtaining the image to be detected.

[0176] S313. Perform virtual object detection on the image to be detected using the detection model, and detect at least one virtual object region.

[0177] Here, at least one virtual object region is the result of virtual object detection corresponding to the image to be detected.

[0178] S314. Perform application processing based on at least one virtual object region.

[0179] In the embodiment of the present application, the detection device extracts features of each virtual object region in the at least one virtual object region, and performs application processing based on the extracted features, wherein the application processing is at least one of recommendation processing, retrieval processing and tracking processing. That is, it can be based on the extracted features to match the retrieval results from the database, to realize the recommendation processing or retrieval processing; or it can be based on the extracted features to determine the motion trajectory of the target virtual object in the video of the virtual scene, to realize the tracking processing. In addition, the application processing can also be other processing based on the virtual object region, which is not limited in the embodiment of the present application.

[0180] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described.

[0181] Referring to Figure 5 , Figure 5 is a flowchart of an exemplary virtual object detection method provided by the embodiment of the present application, wherein the virtual object detection method is executed by a server (detection device); as Figure 5 shown, the exemplary virtual object detection method comprises:

[0182] 1) A real person labeled sample 5-1 (auxiliary image sample and entity object annotation information) is used to train a "Yolov3" model (original auxiliary detection model), to obtain a real person detection model 5-2.

[0183] Here, it should be noted that the target class corresponding to the "Yolov3" model is a real person; referring to Figure 6 , Figure 6 is a network structure diagram of an exemplary real person detection model provided by the embodiment of the present application; as Figure 6 shown, Figure 5 the real person detection model 5-2 in Figure 5 comprises a plurality of convolution layers 6-1, pooling layers 6-2, fully connected layers 6-3 and excitation function layers 6-4; the input data 6-5 is sequentially subjected to the plurality of convolution layers 6-1, the pooling layers 6-2, the fully connected layers 6-3 and the excitation function layers 6-4 in the real person detection model 5-2 in to obtain a real person detection result 6-6; the real person detection result 6-6 is a plurality of human body prediction boxes and corresponding probability values.

[0184] 2) The human body detection model 5-2 is used to detect the human body in the animation unlabeled sample 5-3, to obtain an animation sample 5-4 (initial image sample) from which the human body can be detected from the animation unlabeled sample 5-3.

[0185] Generally speaking, the animation human body (virtual object) similar to the real person picture in the animation unlabeled sample 5-3 can be detected, while the animation human body greatly different from the real person picture is difficult to be detected.

[0186] 3) input the anime sample 5-4 and the real person labeled sample 5-1 into the style model to generate an anime generation sample 5-5 (a transition image sample) with the style information of the anime sample 5-4 and the content information of the real person labeled sample 5-1.

[0187] It should be noted that an image is randomly selected from the anime sample 5-4 as the input information of the style, an image is randomly selected from the real person labeled sample 5-1 as the input information of the content, and the image is input into the style model to output a generation sample, that is, an image in the anime generation sample 5-5. Among them, first, the images in the real person labeled sample 5-1 and the images in the anime sample 5-4 are encoded by using the encoder (for example, a "VGG (Visual Geometry Group)" encoding model) in the style model, to obtain the corresponding features (content features and style features) respectively; then, the style generation module AdaIN (Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization) in the style model maps the mean and variance of the feature distribution of the content to the mean and variance of the feature distribution of the style, to obtain a generation feature (a to-be-decoded image feature); finally, the generation feature is decoded by using the decoder (for example, a "VGG" encoding model) in the style model, so that the image in the anime generation sample 5-5 is obtained.

[0188] Here, the style model is trained, and the process of training the style model is described in detail in the following Figure 7 , Figure 7 is an exemplary style model training process schematic diagram provided by the embodiment of the present application; as shown in Figure 7 , the image 7-1 is a content sample (a content training sample) for training the style model, and the image 7-2 is a style sample (a style training sample) for training the style model; first, the image 7-1 and the image 7-2 pass through the encoder 7-31, the style generation module 7-32 and the decoder 7-33 of the to-be-trained style model 7-3 (an original style model), to obtain the image 7-4 (a predicted style image); then, the content difference 7-51 between the image 7-1 and the image 7-4, and the style difference 7-52 between the image 7-2 and the image 7-4 are used to jointly train the to-be-trained style model 7-3; when the training of the to-be-trained style model 7-3 is completed, the style model is obtained.

[0189] 4) the label of the real person labeled sample 5-1 is used as the label of the anime generation sample 5-5, the real person detection model 5-2 is trained, and the detection model 5-6 (an original detection model) is obtained.

[0190] It should be noted that, since the anime generation sample 5-5 has the same image content as the real person labeled sample 5-1, the anime generation sample 5-5 has the same human body positioning as the real person labeled sample 5-1. See Figure 8 , Figure 8 is an exemplary automatic human body labeling schematic diagram provided by an embodiment of the present application; as shown in Figure 8 , image 8-1 is an image in the real person labeled sample 5-1 in Figure 5 , and the rectangular box 8-11 is the human body positioning box of the image 8-1; the five new style images generated based on the content information of the image 8-1 are image 8-2 to image 8-6, so the human body positioning boxes corresponding to the image 8-2 to image 8-6 are the same as the rectangular box 8-11, and the human body positioning boxes corresponding to the image 8-2 to image 8-6 are rectangular box 8-12 to rectangular box 8-16 in turn. Here, the rectangular box 8-11 to the rectangular box 8-16 are the same rectangular box. That is, the image generated by the style model shares the human body positioning box with the image corresponding to the original content information.

[0191] It should also be noted that, through the content information of the real person field image, with the help of the style information of the anime field image, the new style over-exposed image (anime generation sample 5-5) is obtained to train the real person detection model 5-2, which realizes the increase of the sample quantity, and also effectively utilizes the training samples and models of the real person field, and improves the training efficiency, accuracy and recall rate of the detection model.

[0192] So far, the starting phase of the detection model training has been completed, and through the starting phase, the detection model with initial anime human body detection capability can be obtained under the premise of no anime label. The active labeling iteration phase is described below:

[0193] 5) Perform human body detection (virtual object detection) on the anime unlabeled sample 5-3 by the detection model 5-6, to screen out the target sample 5-7 (to-be-labeled image sample) with labeling value.

[0194] It should be noted that for each image in the animation unlabeled sample 5-3, at least one rectangular prediction box and the probability of the human body in each prediction box (at least one prediction box probability value corresponding to at least one virtual object prediction box) can be detected. Here, first, the number of rectangular prediction boxes with a probability within a certain threshold range in each image in the animation unlabeled sample 5-3 is calculated; then, the images in the animation unlabeled sample 5-3 are sorted based on the number of rectangular prediction boxes; then, the sorted animation unlabeled sample 5-3 is evenly divided into 10 parts (multiple level image samples), corresponding to 10 levels, the number of rectangular prediction boxes of the highest level is the most, and the number of rectangular prediction boxes of the lowest level is the least; finally, since the images in each part are also sorted according to the number of rectangular prediction boxes, in each part, k (number threshold) images are selected in order from high to low according to the number of rectangular prediction boxes, that is, the target sample 5-7 with 10*k images is obtained. It is easy to know that by grading the animation unlabeled sample 5-3, each image is treated differently in the training process of the detection model, and the images (target sample 5-7) that the detection model 5-6 needs to pay attention to most can be selected for labeling, so that the labeling result can maximize the recognition ability of the detection model.

[0195] 6) Labeling the target sample 5-7.

[0196] It should be noted that the target sample 5-7 can be manually labeled by using a labeling tool. Referring to Figure 9 , Figure 9 is an exemplary labeling tool page diagram provided by the embodiment of the present application; as shown in Figure 9 , the page 9-1 is a labeling page of a visual labeling tool; the area 9-11 is used to display the image to be labeled, the area 9-12 is used to display the image category, and the area 9-13 is used to display the file name list corresponding to the target sample 5-7. The labeling personnel label the images in the target sample 5-7 in the page 9-1 one by one, so as to label the position of the human body in each image of the target sample 5-7 in the page 9-1 (for example, a rectangular box, the human body is in the rectangular box, and the position information of the rectangular box is represented by two points of the upper left corner (x1, y1) and the lower right corner (x2, y2)). Figure 5 Figure 5 7) The animation sample 5-8 (remaining image sample) in the animation unlabeled sample 5-3 is taken as the style information, and the target sample 5-7 is taken as the content information, and the animation generation sample 5-9 (incremental sample) is generated by using the style model.

[0197] 7) The animation sample 5-8 (remaining image sample) in the animation unlabeled sample 5-3 is taken as the style information, and the target sample 5-7 is taken as the content information, and the animation generation sample 5-9 (incremental sample) is generated by using the style model.

[0198] ​8) The labeled label (virtual object labeling information) of the target sample 5-7 is taken as the label of the animation generation sample 5-9, the detection model 5-6 is trained, and the detection model 5-10 (trained detection model) is obtained.

[0199] 9) Continue to select samples with labeling value from the animation sample 5-8 by the detection model 5-10, and repeat 5) to 8), until the detection model reaches the expected detection rate for the specified test set (test data set) human body detection, or the labeled data reaches the preset labeling amount, the labeling iteration training process is settled, and the final detection model 5-11 is obtained.

[0200] Here, every time new labeling data is obtained for the images in the animation unlabeled sample 5-3, a batch of animation generation samples with true value labeling data is generated for the unlabeled images by the style model, thereby improving the recall rate of the detection model for animation human body detection.

[0201] The following continues to illustrate an exemplary structure of the implementation of the virtual object detection apparatus 455 as a software module provided in an embodiment of the application. In some embodiments, as shown in Figure 2 The software module stored in the virtual object detection apparatus 455 in the memory 450 can include:

[0202] The sample selection module 4551 is configured to obtain image samples and select image samples to be labeled from the image samples, thereby obtaining remaining image samples that are not selected, wherein each sample image in the image samples includes a virtual object region.

[0203] The style conversion module 4552 is configured to combine the style information of the remaining image samples with the content information of the image samples to be labeled to obtain incremental samples.

[0204] The model training module 4553 is configured to obtain virtual object labeling information of the image samples to be labeled, and determine the virtual object labeling information as label information of the incremental samples and the image samples to be labeled, and train the original detection model.

[0205] The model iteration module 4554 is configured to continue to select new image samples to be labeled from the remaining image samples to generate new incremental samples, and train the trained original detection model based on the new incremental samples, until a training termination condition is met, and obtain a detection model, wherein the detection model is used for virtual object detection on images.

[0206] In the embodiment of the present application, the sample selection module 4551 is further configured to divide the image samples into multiple levels based on image difference information, wherein the image difference information is at least one of virtual object quantity information, image quality information, light information, virtual object position information, view angle information, and virtual style information; and combine a quantity threshold of sample images selected from each level of image samples into the to-be-labeled image samples of the image samples.

[0207] In the embodiment of the present application, when the image difference information is the virtual object quantity information, the sample selection module 4551 is further configured to obtain the original detection model; perform virtual object detection on each sample image by using the original detection model to obtain at least one prediction box probability value corresponding to at least one virtual object prediction box, wherein each virtual object prediction box in the at least one virtual object prediction box is used to label the virtual object region; compare the at least one prediction box probability value with a probability threshold respectively; determine a prediction box quantity corresponding to the at least one virtual object prediction box based on a comparison result; and determine the prediction box quantity as the virtual object quantity information.

[0208] In the embodiment of the present application, the sample selection module 4551 is further configured to calculate a probability entropy corresponding to the at least one prediction box probability value; and determine the probability entropy as the virtual object quantity information.

[0209] In the embodiment of the present application, the sample selection module 4551 is further configured to obtain entity object labeling information corresponding to auxiliary image samples, wherein each auxiliary sample image in the auxiliary image samples includes an entity object region; train an auxiliary detection model based on the auxiliary image samples and the entity object labeling information, wherein the auxiliary detection model is used to perform entity object detection on images; perform virtual object detection on each sample image by using the auxiliary detection model to obtain an initial detection result; combine sample images with at least one virtual object detection box in the image samples into initial image samples; combine content information of the auxiliary image samples and style information of the initial image samples to obtain transition image samples; and train the auxiliary detection model based on the transition image samples and the entity object labeling information to obtain the original detection model.

[0210] In the embodiment of the present application, when the image difference information is the image quality information, the sample selection module 4551 is further configured to obtain an image quality score of each sample image in the image samples; and determine the image quality score as the image quality information corresponding to each sample image.

[0211] In the embodiment of the present application, the model training module 4553 is further configured to perform virtual object detection on each of the incremental samples and the image samples to be labeled by using the original detection model, to obtain at least one prediction probability value corresponding to at least one virtual object frame; obtain at least one annotation frame corresponding to each of the samples from the virtual object annotation information; determine a virtual object detection loss based on the at least one prediction probability value corresponding to the at least one virtual object frame and the at least one annotation frame; and perform back propagation in the original detection model based on the virtual object detection loss to train the original detection model.

[0212] In the embodiment of the present application, the style conversion module 4552 is further configured to perform feature extraction on the image samples to be labeled to obtain content features, and perform feature extraction on the remaining image samples to obtain style features; perform distribution calculation on each channel style feature in the style features to obtain style feature distribution information, and perform distribution calculation on each channel content feature in the content features to obtain content feature distribution information; map the content feature distribution information to the style feature distribution information to obtain to-be-decoded image features; and decode the to-be-decoded image features to obtain the incremental samples.

[0213] In the embodiment of the present application, the feature extraction, the distribution calculation, the mapping and the decoding are implemented by using a style model; the detection device 455 further includes a style model training module 4555 configured to initialize model parameters of an original style model; fuse a style training sample and a content training sample by using the original style model to obtain a predicted style image; perform back propagation in the original style model based on a style difference between the style training sample and the predicted style image and a content difference between the content training sample and the predicted style image to obtain the style model.

[0214] In the embodiment of the present application, the detection device 455 further includes a condition judgment module 4556 configured to obtain a current trained original detection model; calculate a detection rate of the current trained original detection model based on a test data set, wherein the detection rate includes at least one of an accuracy rate and a recall rate; and determine that the training stop condition is met when the detection rate is greater than a detection rate threshold.

[0215] In the embodiment of the present application, the condition judgment module 4556 is further configured to obtain a number of annotated sample images in the image samples; calculate an annotation proportion of the number of annotated sample images and a sample number of the image samples; and determine that the training stop condition is met when the annotation proportion is greater than a proportion threshold.

[0216] In the embodiment of the present application, the detection apparatus 455 further comprises an object detection module 4557 configured to acquire a virtual object detection request, acquire a to-be-detected image in response to the virtual object detection request, perform virtual object detection on the to-be-detected image by using the detection model, and detect at least one virtual object region. The application processing is based on the at least one virtual object region, and the application processing is at least one of recommendation processing, retrieval processing, and tracking processing.

[0217] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the virtual object detection method provided in the embodiment of the present application.

[0218] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, cause the processor to execute the virtual object detection method provided in the embodiment of the present application, for example, as shown in the virtual object detection method. Figure 3 The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, cause the processor to execute the virtual object detection method provided in the embodiment of the present application, for example, as shown in the virtual object detection method.

[0219] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM, etc. memory; or various devices comprising one or any combination of the above memories.

[0220] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0221] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0222] As an example, the executable instructions can be deployed in one computer device (at this time, the one computer device is the detection device), or in multiple computer devices located in one place (at this time, the multiple computer devices located in one place are the detection devices), or in multiple computer devices distributed in multiple places and interconnected through a communication network (at this time, the multiple computer devices distributed in multiple places and interconnected through a communication network are the detection devices).

[0223] To sum up, through the embodiments of the present application, the image samples to be labeled are selected from the image samples for labeling, and the incremental samples are generated based on the style information of the remaining image samples that are not selected, and the labeled information is used as the label common to the image samples to be labeled and the incremental samples, and the original detection model is trained; then, based on the remaining image samples, the above-mentioned selective labeling, sample generation and training process are cyclically executed until the final detection model is obtained; in this way, since the incremental samples contain the style information of the remaining image samples in the training samples used in the detection model acquisition process, when the incremental samples and the selected samples are used to perform model training, and the above-mentioned selective labeling, sample generation and training process are cyclically executed based on the remaining image samples, the overall information of the image samples can be integrated to realize effective training of the model, and then the learning ability of the model can be improved and the accuracy of virtual object detection can be improved in the case of limited training samples. In addition, by differentiating the sample images, more representative samples can be selected for labeling, and the training efficiency of the model can be improved; and by style conversion to expand the number of samples, the recall rate and detection accuracy of the detection model can be improved.

[0224] The above only describes the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for virtual object detection, characterized in that, The method comprises the following steps: obtaining image samples, and classifying the image samples based on image difference information to obtain a plurality of classified image samples, wherein the image difference information is at least one of virtual object quantity information, image quality information, light information, virtual object position information, view angle information and virtual style information; selecting a threshold number of sample images from each of the plurality of classified image samples, and combining the selected sample images into a to-be-labeled image sample of the image sample, thereby obtaining a remaining image sample that is not selected, wherein each sample image in the image sample comprises a virtual object region; combining style information of the remaining image sample with content information of the to-be-labeled image sample to obtain an incremental sample; obtaining virtual object labeling information of the to-be-labeled image sample, and determining the virtual object labeling information as label information of the incremental sample and the to-be-labeled image sample, and training an original detection model; continuing to select a new to-be-labeled image sample from the remaining image sample to generate a new incremental sample, and training the trained original detection model based on the new incremental sample, until a training termination condition is met, thereby obtaining a detection model, wherein the detection model is used for virtual object detection of an image.

2. The method of claim 1, wherein, When the image difference information is the virtual object quantity information, before classifying the image samples based on the image difference information, the method further comprises: obtaining the original detection model; performing virtual object detection on each sample image through the original detection model to obtain at least one prediction box probability value corresponding to at least one virtual object prediction box, wherein each virtual object prediction box in the at least one virtual object prediction box is used for labeling the virtual object region; comparing the at least one prediction box probability value with a probability threshold value respectively; determining a prediction box quantity corresponding to the at least one virtual object prediction box based on the comparison result; determining the prediction box quantity as the virtual object quantity information.

3. The method of claim 2, wherein, After obtaining the at least one prediction box probability value corresponding to the at least one virtual object prediction box, the method further comprises: calculating a probability entropy corresponding to the at least one prediction box probability value; determining the probability entropy as the virtual object quantity information.

4. The method according to claim 2 or 3, characterized in that, The method of obtaining the original detection model comprises: obtaining entity object labeling information corresponding to auxiliary image samples, wherein each auxiliary sample image in the auxiliary image samples comprises an entity object region; training an auxiliary detection model based on the auxiliary image samples and the entity object labeling information, wherein the auxiliary detection model is used for entity object detection of an image; performing virtual object detection on each sample image through the auxiliary detection model to obtain an initial detection result; combining sample images with at least one virtual object detection box in the initial detection result into an initial image sample in the image samples; combining content information of the auxiliary image samples with style information of the initial image sample to obtain a transition image sample; Train the auxiliary detection model based on the transition image sample and the entity object annotation information to obtain the original detection model.

5. The method of claim 1, wherein, When the image difference information is the image quality information, before the image sample is classified into different levels based on the image difference information, the method further comprises: Obtain the image quality score of each sample image in the image sample; Determine the image quality score as the image quality information corresponding to each sample image.

6. The method according to any one of claims 1 to 3, characterized in that, The determination of the virtual object annotation information as the label information of the incremental sample and the image sample to be annotated, and the training of the original detection model, comprises: Perform virtual object detection on each sample in the incremental sample and the image sample to be annotated through the original detection model to obtain at least one prediction probability value corresponding to at least one virtual object frame; Obtain at least one annotation frame corresponding to each sample from the virtual object annotation information; Determine a virtual object detection loss based on the at least one prediction probability value corresponding to the at least one virtual object frame and the at least one annotation frame; Perform back propagation in the original detection model based on the virtual object detection loss to train the original detection model.

7. The method according to any one of claims 1 to 3, characterized in that, The combination of the style information of the remaining image sample and the content information of the image sample to be annotated to obtain an incremental sample, comprises: Perform feature extraction on the image sample to be annotated to obtain content features, and perform feature extraction on the remaining image sample to obtain style features; Perform distribution calculation on each channel style feature in the style features to obtain style feature distribution information, and perform distribution calculation on each channel content feature in the content features to obtain content feature distribution information; Map the content feature distribution information to the style feature distribution information to obtain a to-be-decoded image feature; Decode the to-be-decoded image feature to obtain the incremental sample.

8. The method of claim 7, wherein, The feature extraction, the distribution calculation, the mapping, and the decoding are realized through a style model; Before the feature extraction on the image sample to be annotated, the method further comprises: Train the style model by the following way: Initialize the model parameters of an original style model; Fuse the style training sample and the content training sample through the original style model to obtain a predicted style image; Perform back propagation in the original style model based on the style difference between the style training sample and the predicted style image, and the content difference between the content training sample and the predicted style image to obtain the style model.

9. The method according to any one of claims 1 to 3, characterized in that, Before the detection model is obtained when the training stop condition is met, the method further comprises: Obtain a currently trained original detection model; Calculate the detection rate of the currently trained original detection model based on a test data set, wherein the detection rate comprises at least one of the accuracy and the recall rate; When the detection rate is greater than a detection rate threshold, it is determined that the training stop condition is met.

10. The method according to any one of claims 1 to 3, characterized in that, Before the detection model is obtained when the training stop condition is met, the method further comprises: obtain a number of labeled sample images in the image samples; calculate a labeling proportion of the number of labeled sample images and a number of samples of the image samples; determine that the training stopping condition is met when the labeling proportion is greater than a proportion threshold.

11. The method according to any one of claims 1 to 3, characterized in that, After the detection model is obtained, the method further includes: obtaining a virtual object detection request; obtaining an image to be detected in response to the virtual object detection request; detecting at least one virtual object region through the detection model for the image to be detected; performing application processing based on the at least one virtual object region, wherein the application processing is at least one of recommendation processing, retrieval processing, and tracking processing.

12. A virtual object detection apparatus characterized by comprising: comprise: a sample selection module configured to obtain image samples, and perform hierarchical division on the image samples based on image difference information to obtain a plurality of hierarchical image samples, wherein the image difference information is at least one of virtual object quantity information, image quality information, lighting information, virtual object position information, view angle information, and virtual style information; combine a number threshold of sample images selected from each hierarchical image sample in the plurality of hierarchical image samples into labeled image samples of the image samples, thereby obtaining remaining image samples that are not selected, wherein each sample image in the image samples comprises a virtual object region; a style conversion module configured to obtain incremental samples by combining style information of the remaining image samples and content information of the labeled image samples; a model training module configured to obtain virtual object labeling information of the labeled image samples, and determine the virtual object labeling information as label information of the incremental samples and the labeled image samples to train an original detection model; a model iteration module configured to continue to select new labeled image samples from the remaining image samples to generate new incremental samples, and train the original detection model after training based on the new incremental samples, until a detection model is obtained when a training stopping condition is met, wherein the detection model is used for virtual object detection on images.

13. The apparatus of claim 12, wherein, When the image difference information is the virtual object quantity information, the sample selection module is further configured to obtain the original detection model; perform virtual object detection on each sample image through the original detection model to obtain at least one prediction box probability value corresponding to at least one virtual object prediction box, wherein each virtual object prediction box in the at least one virtual object prediction box is used for labeling the virtual object region; compare the at least one prediction box probability value with a probability threshold respectively; determine a number of prediction boxes corresponding to the at least one virtual object prediction box based on a comparison result; and determine the number of prediction boxes as the virtual object quantity information.

14. The apparatus of claim 13, wherein the sample selection module is further configured to calculate a probability entropy corresponding to the at least one prediction box probability value; and determine the probability entropy as the virtual object quantity information.

15. The apparatus of claim 13 or 14, wherein The sample selection module is further configured to: obtain entity object label information corresponding to auxiliary image samples, wherein each auxiliary sample image in the auxiliary image samples comprises an entity object region; train an auxiliary detection model based on the auxiliary image samples and the entity object label information, wherein the auxiliary detection model is configured to perform entity object detection on images; perform virtual object detection on each sample image by using the auxiliary detection model to obtain an initial detection result; combine sample images with at least one virtual object detection box in the initial detection result in the image samples as initial image samples; combine content information of the auxiliary image samples and style information of the initial image samples to obtain transition image samples; and train the auxiliary detection model based on the transition image samples and the entity object label information to obtain the original detection model.

16. The apparatus of claim 12, wherein, When the image difference information is the image quality information, the sample selection module is further configured to: obtain an image quality score of each sample image in the image samples; and determine the image quality score as the image quality information corresponding to each sample image.

17. The apparatus of any one of claims 12-14, wherein, The model training module is further configured to: perform virtual object detection on each sample in the incremental samples and the image samples to be labeled by using the original detection model to obtain at least one prediction probability value corresponding to at least one virtual object box; and obtain at least one label box corresponding to each sample from the virtual object label information; determine a virtual object detection loss based on the at least one prediction probability value corresponding to the at least one virtual object box and the at least one label box; perform back propagation in the original detection model based on the virtual object detection loss to train the original detection model.

18. The apparatus of any one of claims 12-14, wherein, The style conversion module is further configured to: perform feature extraction on the image samples to be labeled to obtain content features, and perform feature extraction on the remaining image samples to obtain style features; perform distribution calculation on each channel style feature in the style features to obtain style feature distribution information, and perform distribution calculation on each channel content feature in the content features to obtain content feature distribution information; map the content feature distribution information to the style feature distribution information to obtain to-be-decoded image features; and decode the to-be-decoded image features to obtain the incremental samples.

19. The apparatus of claim 18, wherein, The feature extraction, the distribution calculation, the mapping and the decoding are implemented by a style model; the detection device further comprises a style model training module, configured to initialize model parameters of an original style model; the style model training module is further configured to fuse a style training sample and a content training sample through the original style model to obtain a predicted style image; the style model training module is further configured to perform back propagation in the original style model based on a style difference between the style training sample and the predicted style image and a content difference between the content training sample and the predicted style image to obtain the style model.

20. The apparatus of any one of claims 12 to 14, wherein, The detection device further comprises: The condition judging module is configured to obtain a current trained original detection model; calculate a detection rate of the current trained original detection model based on a test data set, wherein the detection rate comprises at least one of an accuracy rate and a recall rate; and determine that the training stop condition is met when the detection rate is greater than a detection rate threshold.

21. The apparatus of claim 20, wherein The condition judging module is further configured to obtain a number of labeled sample images in the image sample; calculate a labeling proportion of the number of labeled sample images and a sample number of the image sample; and determine that the training stop condition is met when the labeling proportion is greater than a proportion threshold.

22. The apparatus of any one of claims 12 to 14, wherein, The detection device further comprises: The object detection module is configured to obtain a virtual object detection request; obtain a to-be-detected image in response to the virtual object detection request; perform virtual object detection on the to-be-detected image through the detection model to detect at least one virtual object region; and perform application processing based on the at least one virtual object region, wherein the application processing is at least one of recommendation processing, retrieval processing and tracking processing.

23. A virtual object detection device, comprising: Comprise: A memory configured to store executable instructions; A processor configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, Executable instructions are stored, and when executed by a processor, the method of any one of claims 1 to 11 is implemented.

25. A computer program product, comprising: Computer instructions, wherein the computer instructions are executed by a processor to implement the method of any one of claims 1 to 11.

Images