Model training method, scene recognition method, computing device and medium

Abstract

The present disclosure provides a model training method, a scene recognition method, a computing device and a medium. The training method includes: obtaining N prototype images and a video library for model training, wherein the N prototype images contain the target scene, and the video library has identification information associated with the target scene, wherein N is an integer greater than 1; obtaining a video image library based on the video library, wherein the video image library includes multiple video images; based on the N prototype images and the distribution characteristics of the multiple video images, selecting a training sample set for a target scene recognition model from the multiple video images, wherein the training sample set includes a positive sample set containing the target scene and a negative sample set not containing the target scene; and using the training sample set to train the target scene recognition model.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a training method for a target scene recognition model, a target scene recognition method, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of information technology, users are increasingly transmitting information through videos, which can involve various scenes, such as sports scenes, food scenes, portrait scenes, landscape scenes, and cartoon scenes.

[0003] At present, it is possible to identify whether a video contains a target scene (for video images extracted from the video) through machine learning models based on artificial intelligence (e.g., deep learning models, convolutional neural networks). When the training sample set comes from an initial video library (i.e., a collection of various massive videos, which can be understood as a full video library), for example, in the case where the target scene is a "shooting" scene, a large number of training samples in the video image library of the initial video library can be manually annotated, including training samples containing shooting scenes, and used to train the machine learning model, so that the trained model is used to identify whether the video to be identified contains images of the shooting moment. However, this method has the need to annotate massive training samples, and is not conducive to migration to other scene categories, because when identifying new scenes, it is also necessary to annotate massive samples, which leads to the complexity of the overall algorithm design, excessive manual investment, excessive requirements on the ability of annotation personnel, and very slow scene iteration.

[0004] Therefore, how to simply, quickly and accurately obtain a large number of training samples for the target scene in order to obtain a more effective target scene recognition model is an urgent problem to be solved. Summary of the invention

[0005] According to one aspect of the present disclosure, a training method for a target scene recognition model is provided, comprising: obtaining N prototype images and a video library for model training, the N prototype images containing a target scene, the videos in the video library having identification information associated with the target scene, wherein N is an integer greater than 1; obtaining a video image library based on the video library, the video image library including a plurality of video images; selecting a training sample set for a target scene recognition model from the plurality of video images based on the distribution characteristics of the N prototype images and the plurality of video images, the training sample set including a positive sample set containing the target scene and a negative sample set not containing the target scene; and training the target scene recognition model using the training sample set.

[0006] According to another aspect of the present disclosure, a method for identifying a target scene is provided, comprising: acquiring a video to be identified; generating a video image library to be identified based on the video to be identified; using a target scene recognition model trained by the training method described above to identify whether a video image in the video image library to be identified contains a target scene; and determining that the video to be identified contains the target scene when at least one video image in the video image library to be identified contains the target scene.

[0007] According to another aspect of the present disclosure, a training device for a target scene recognition model is provided, comprising a first acquisition module, a first image library generation module, a sample selection module and a training module. The first acquisition module is used to acquire N prototype images and a video library for model training, wherein the N prototype images contain the target scene, and the videos in the video library have identification information associated with the target scene, wherein N is an integer greater than 1; the first image library generation module is used to obtain a video image library based on the video library, wherein the video image library includes a plurality of video images; the sample selection module is used to select a training sample set for the target scene recognition model from the plurality of video images based on the distribution characteristics of the N prototype images and the plurality of video images, wherein the training sample set includes a positive sample set containing the target scene and a negative sample set not containing the target scene; the training module is used to train the target scene recognition model using the training sample set.

[0008] According to another aspect of the present disclosure, a target scene recognition device is provided, comprising a second acquisition module, a second image library generation module, a recognition module and a determination module. The second acquisition module is used to acquire a video to be recognized; the second image library generation module is used to generate a video image library to be recognized based on the video to be recognized; the recognition module is used to use a target scene recognition model to recognize whether a video image in the video image library to be recognized contains a target scene, wherein the target scene recognition model can be local or obtained from the outside, and the target scene recognition model is trained according to the training method described above; the determination module is used to determine that the video to be recognized contains the target scene when at least one video image in the video image library to be recognized contains the target scene.

[0009] According to an embodiment of the present disclosure, the target scene recognition device may also include a manual review module, which is used to receive a review result of at least one of the multiple videos to be recognized; when the recognition result for any video to be recognized indicates that it does not contain the target scene but its corresponding review result indicates that it contains the target scene, the video image containing the target scene in the video image library of the video to be recognized is used as a new prototype image for selecting a new training sample set for the target scene recognition model.

[0010] According to another aspect of the present disclosure, a computing device is provided. The computing device includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, each step of the method described above is implemented.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes each step of the method described above.

[0012] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, each step of the method described above is implemented.

[0013] Based on the above aspects of the present disclosure, in an example of using an existing initial video library to generate a training sample set for training a target scene recognition model, multiple positive and negative samples can be automatically mined from the video image library of the video library based on the distribution characteristics of a small number of prototype images of the target scene and a video library associated with the target scene (obtained from the initial video library based on identification information), thereby effectively eliminating irrelevant video images, thereby greatly reducing the samples to be labeled. In addition, when the target scene to be identified is changed, only the prototype image needs to be changed and the keywords of the video library need to be filtered, that is, this method of generating a training sample set for training a target scene recognition model can be directly migrated to other target scene recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A A schematic diagram of a training process for a target scene recognition model is shown.

[0015] Figure 1B A network architecture diagram of a target scene recognition model training method and a target scene recognition method according to an embodiment of the present disclosure is shown.

[0016] Figure 1C Shows Figure 1B Schematic diagram of the interaction between the server and the external environment in the network architecture.

[0017] Figure 1D A schematic diagram showing a structure of a network architecture provided by an embodiment of the present disclosure applied to a blockchain system;

[0018] Figure 2 A flow chart of a method for training a target scene recognition model according to an embodiment of the present disclosure is shown.

[0019] Figure 3The model structure diagram of the deep residual network 101 layer (RESNET-101) is shown.

[0020] Figures 4A-4C More details of the step of selecting a training sample set according to the training method of an embodiment of the present disclosure are shown.

[0021] Figure 5 A schematic diagram showing the result of selecting a training sample set (mining out positive and negative samples) from the multiple video images in the video image library is shown.

[0022] Figure 6 The distribution of positive samples mined from a video library based on 6 example videos is shown.

[0023] Figure 7 More details of training the target scene recognition model using a training sample set including noise samples are shown.

[0024] Figures 8A-8B A schematic flow chart of a method for identifying a target scene according to an embodiment of the present disclosure is shown.

[0025] Fig. 9 A schematic diagram of interactions involved in a target scene recognition method according to an embodiment of the present disclosure with a manual verification step added is shown.

[0026] Figures 10A-10B A structural block diagram of a training device 1000 for a target scene recognition model is shown.

[0027] Fig.11 A structural block diagram of a target scene recognition device 1100 is shown.

[0028] Fig.12 A structural block diagram of a computing device 1200 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0030] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0031] Before the detailed description of the present disclosure is given, some terms that may be used in the present disclosure may be briefly explained.

[0032] Image recognition: It belongs to the category level recognition. It does not consider the specific instance of the object, but only the category of the object (such as people, dogs, cats, birds, etc.). The output is the category to which the object belongs. A typical recognition task is the recognition task in the large-scale general object recognition open source dataset ImageNet, which identifies which of the 1,000 categories an object belongs to. Image recognition provides semantic information of the image.

[0033] Imagenet: A large open-source dataset for general object recognition.

[0034] RESNET-101: A deep residual network model that has shown good test results in various image classification tasks.

[0035] Image recognition and annotation: Annotate one image at a time and indicate that the image belongs to a certain category in the specified category.

[0036] Multiple instance learning: Multiple instance learning (MIL) is an evolution of supervised learning. Instead of inputting a series of individually annotated examples, in multiple instance learning, the input is a series of annotated "bags", each of which contains many examples. In multiple instance learning, an image is described as a bag: X = {X1, X2, ..., Xm}, where each Xi is a feature vector extracted from the corresponding i-th region of the image (which we call an example), and m is the number of regions (examples) into which the image is segmented, 0 ≤ i ≤ m. When an image bag contains both "sand" region examples and "water" region examples, the bag is labeled as a positive example ("beach").

[0037] Metric Learning: Metric Learning can be considered as Distance Metric Learning (DML) or Similarity Learning. In mathematics, a metric (or distance function) is a function that defines the distance between elements in a set. Metric Learning is a commonly used traditional machine learning method. Its basic principle is to autonomously learn a metric distance function for a specific task based on different tasks. Metric Learning has a good classification effect in the fields of face recognition and text.

[0038] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0039] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0040] As mentioned above, when using the existing initial video library to generate a training sample set for training the target scene recognition model, a large number of training samples will be obtained by extracting video images from the video. For example, an average of 3 minutes of short video (assuming 25fps, there are 25*60*3=4500 video images). If the initial video library has 1 million videos, there are a total of 4.5 billion video images to be labeled. The sample size is extremely large. If based on the current manual labeling method, it will require a huge amount of labeling resources, and for scenes with ambiguous samples, the labeling difficulty increases. At the same time, scene iteration is also very slow, and different scenes will result in more manual investment, which is not conducive to rapid business expansion.

[0041] like Figure 1A As shown, firstly, all video images are collected, and then the annotators manually annotate all video images. After all the annotated video images are used for model training, a trained model is obtained. Then, the model is tested using the test set, and the test results can be fed back to adjust the model parameters, and data problems can be fed back to the annotators to adjust the annotations. In addition, the various indicators of the model are provided to the product evaluators to determine whether the preset indicators are met.

[0042] In the real video understanding task, a common problem is how to filter out videos with certain target scenes (such as basketball shooting moments) from a large number of videos. Generally, when users upload videos, they will be accompanied by a series of identification information such as titles, descriptions, and tags. Therefore, in order to eliminate some videos that are extremely irrelevant to the target scene, the technical solution disclosed in the present invention proposes that first, based on the keywords of the target scene, a video library with identification information associated with the target scene can be filtered out from the initial video library, and then based on the distribution characteristics of multiple video images in the video image library of the video library and a small number of prototype images containing the target scene, some video images are selected from the video image library as a training sample set, and then the target scene recognition model is trained using the training sample set. Therefore, irrelevant video images can be effectively eliminated, and the samples to be annotated can be greatly reduced. In addition, when the target scene to be identified is changed, only the prototype image and the keywords of the video library need to be changed, that is, this method of generating training samples for training the target scene recognition model can be directly migrated to other target scene recognition tasks.

[0043] Figure 1B A network architecture diagram of a target scene recognition model training method and a target scene recognition method according to an embodiment of the present disclosure is shown. Figure 1C Shows Figure 1B Schematic diagram of the interaction between the server and the external environment in the network architecture.

[0044] The target scene recognition model training method and scene recognition method of the disclosed embodiment can be applied to the following: Figure 1B In the network architecture shown in Figure 1B As shown, the user terminals 100 are connected to the servers 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and a wired or wireless link is used to realize data transmission.

[0045] Recombination Figure 1B and 1C, the server 200 can obtain the prototype image and the initial video library by locally storing the prototype image and a large amount of initial video library or obtaining a network link of the prototype image and the initial video library from the outside (such as the user terminal 100). After obtaining the prototype image and the initial video library, the server 200 can also receive the keywords of the target scene (for example, "shooting") from its own input device or from the user terminal, and then based on the received keywords of the target scene, it can filter out videos with identification information associated with the target scene (for example, at least one of the title, label, description, and keywords) from the initial video library, thereby forming a video library for generating training samples. For example, if the target scene is "shooting", then the identification information associated with the target scene can be a title, description, label, etc. including keywords such as "basketball", "shooting", "women's basketball", "jump shot", "three-point shot", etc. The server can select a video library for generating training samples based on a small number of prototype images (such as Figure 1B The target scene recognition model is trained by using the several prototype images of the target scene including "shooting" in the video library and the distribution characteristics of multiple video images in the video image library of the video library obtained by screening based on the identification information, and multiple positive and negative samples are selected (mined) from the video library as training samples to train the target scene recognition model, and store (for local recognition tasks) or send the trained target scene recognition model to other processing devices (for recognition tasks of other processing devices). The present disclosure takes the example of a server storing the trained target scene recognition model locally, but obviously, the server can also send the trained target scene recognition model to other processing devices (other servers) for other processing devices to perform target scene recognition.

[0046] The user terminal 100 can also send the video to be identified to the server 200. The server extracts frames from the video to be identified received from the user terminal to generate a video image library to be identified, and uses the trained target scene recognition model to perform target scene recognition on the video image library to determine whether there is a video image containing the target scene in the video image library to be identified, and then determines whether the video to be identified contains the target scene and returns the recognition result to the user terminal. Figure 1C As shown, four video images in video 1 input by the user contain the target scene, while no video image in video 2 contains the target scene, so a "yes" result is returned for video 1, and a "no" result is returned for video 2.

[0047] exist Figure 1BWhat is shown is that the training method and the target scene recognition method of the target scene recognition model are executed at the server (the specific operation will be described later), but it should be understood that if the user terminal itself has sufficient computing and processing capabilities, and can obtain enough initial video libraries and prototype images and videos to be recognized, then the training method and the target scene recognition method of the target scene recognition model can also be executed at the user terminal. In addition, the server and the user terminal can also respectively execute part of these methods, thereby executing these methods together.

[0048] In the present disclosure, a user terminal may be, for example, a mobile device, a computer terminal, or other device with computing and processing capabilities and communication capabilities that can be used by a user.

[0049] In the description of the present disclosure, for convenience, servers of various possible types are collectively referred to as servers. Therefore, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms.

[0050] In addition, the server can also interact with the client to achieve Figure 1B The above interactive function between the client and the user terminal shown in (the server executes the above training method and target scene recognition method). The client is a client of a specific application, and the specific application can realize the above Figure 1B The operations described in the user terminal 100 interacting with the server 200.

[0051] The trained target scene recognition model proposed in the embodiments of the present disclosure can be deployed in a cloud server, which provides artificial intelligence cloud services. The so-called artificial intelligence cloud services are generally referred to as AIaaS (AI asa Service, which means "AI as a service" in Chinese). This is a mainstream service mode of an artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud.

[0052] In other application scenarios, clients (for example, video application clients) and multiple nodes (any form of computing devices connected to the network, such as servers and user terminals) are connected through network communication to form a distributed system. Figure 1B The user terminal 100 in the embodiment, and the computing device at the node communicating with the client can be Figure 1BServer 200 in.

[0053] In addition, a user terminal with sufficient computing and processing capabilities can also be used as a computing device at a node to execute the target scene recognition model training method and the target scene recognition method in the present disclosure.

[0054] Here we take the distributed system as a blockchain system as an example, see Figure 1D , Figure 1D This is an optional structural diagram of the network architecture 10 provided by the embodiment of the present disclosure applied to the blockchain system, which is formed by multiple nodes 20 (any form of computing devices in the access network, such as servers and user terminals) and clients 30. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In a distributed system, any machine such as a server or a user terminal can join and become a node. The node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0055] See also Figure 1D The functions of each node in the blockchain system shown include:

[0056] 1) Routing: a basic function of a node, used to support communication between nodes.

[0057] In addition to the routing function, the node can also have the following functions:

[0058] 2) Applications are deployed in the blockchain to implement specific businesses based on actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to a temporary block when they successfully verify the source and integrity of the record data.

[0059] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0060] 4) Consensus is a process in the blockchain network used to reach agreement on transactions in a block among multiple nodes involved. The agreed blocks will be appended to the end of the blockchain. The mechanisms for achieving consensus include Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof-of-Stake (DPoS), Proof of Elapsed Time (PoET), etc.

[0061] For example, the client 30 communicates with a computing device at a node, and after sending a prototype image and an initial video library to the computing device at the node, the computing device can reliably share data in the blockchain system, or after training a target scene recognition model at the client 30, the trained recognition model can be shared at the computing devices of each trusted node.

[0062] The following is combined with Figure 2-7 The training method of the target scene recognition model in the embodiment of the present disclosure is described in detail.

[0063] Figure 2 FIG. 2 is a flow chart of a training method 200 for a target scene recognition model according to an embodiment of the present disclosure. The training method can be executed by any computing device with processing capabilities, such as Figure 1B The server (including cloud server) and / or user terminal shown in, or Figure 1D The blockchain system is executed by a computing device at each node.

[0064] In step S210, N prototype images and a video library for model training are obtained, wherein the N prototype images contain the target scene, and the videos in the video library have identification information associated with the target scene, wherein N is an integer greater than 1.

[0065] Optionally, as described above, the identification information is at least one of a title, a description, and a tag, and the videos in the video library are obtained by screening the initial video library based on the identification information based on the keywords of the target scene.

[0066] For example, if the target scene is "shooting", then the processing device of the server can perform semantic analysis on the received keyword "shooting", and use at least one of the keywords "shooting", "basketball", "women's basketball", "jump shot", "three-point shot" and other keywords associated with the keyword "shooting" to filter the identification information (title, description, tag, etc.) of each video in the initial video library, so as to filter out videos with the identification information of these keywords as the video library of the target scene. This step can reduce the video search range for the target scene from the massive database by several orders of magnitude; of course, if the volume of the initial video library is large enough, semantic expansion can be omitted, and "shooting" can be used directly as a keyword to search for videos. However, these videos may only be associated with "shooting", but not necessarily contain the target scene "shooting". For example, the videos filtered out by the keyword "women's basketball" may only be videos of women's basketball charity activities, and do not contain the target scene "shooting". The videos filtered out by the keyword "shooting" may only be a short video of the audience celebrating after a successful shot. Therefore, these videos (video images) should not be used for training (they are inherently ambiguous).

[0067] Optionally, the prototype image refers to an image that has been determined to contain the target scene. Figure 1B The several prototype images shown are all images containing the target scene "shooting a basketball".

[0068] In the embodiments of the present disclosure, only a small number of prototype images with a limited amount of data are needed to select a training sample set including a sufficient number of video images from a video image library. This process can also be considered as mining training sample sets in the video image library.

[0069] In step S220, a video image library is obtained based on the video library, and the video image library includes a plurality of video images.

[0070] Optionally, each video in the video library can be framed, that is, all video images constituting each video are saved, thereby obtaining multiple video images as a video image library. Subsequent steps require mining video images containing target scenes as positive samples from the video image library, and obtaining video images without target scenes as negative samples.

[0071] In step S230, based on the distribution characteristics of the N prototype images and the multiple video images, a training sample set for a target scene recognition model is selected from the multiple video images, and the training sample set includes a positive sample set containing the target scene and a negative sample set not containing the target scene.

[0072] In this step, the distribution characteristics of the multiple video images may be clustering characteristics of the multiple video images. N prototype images may be used as a benchmark to take video images with sufficiently large similarity to the prototype images in the multiple video images as positive samples, and video images with sufficiently small similarity to the prototype images as negative samples. Therefore, it can also be considered that positive and negative samples are mined from the multiple video images in the video image library based on the prototype images, and these positive and negative samples are obtained by diffusion of these prototype images.

[0073] Optionally, the similarity between images described in this article can be understood as the distance between images, more specifically the distance between feature vectors of images, including Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0074] More details of step S230 will be referred to later. Figure 4A-4B and Figure 5 Describe in detail.

[0075] In step S240, the target scene recognition model is trained using the training sample set.

[0076] Optionally, the target scene recognition model provided by the embodiment of the present disclosure may be a deep neural network model, and the deep neural network model may have a multi-layer neural network structure. For example, the deep neural network model may be a network model based on ResNet101 (deep residual network 101 layer), and the model structure of the deep residual network 101 layer is as follows: Figure 3 As shown in the figure, this is a three-layer residual module used to reduce the number of parameters. 3x3 represents the size of the convolution kernel, and 64 represents the number of channels. A plus sign in a circle represents addition, i.e., identity mapping. ReLU (Rectified Linear Unit) represents activation using an activation function. 256-d represents that the input is 256-dimensional.

[0077] As shown in Table 1, it is a structure table of ResNet101 in the deep neural network model in some embodiments, where x3, x4 and x23 represent 3 blocks, 4 blocks and 23 blocks respectively. There are 5 types of convolutional layers, and Conv5_x is the 5th convolutional layer. As shown in Table 2, it is a structure table of the output layer in the deep neural network model.

[0078] Among them, Conv5_x outputs the deep feature feature map of the training sample image (the feature map obtained after the image and the filter are convolved), the pool_cr layer outputs the image feature vector, which is the deep high-dimensional feature output of the image after the deep learning neural network forward calculation, and can be a one-dimensional feature vector obtained after the pooling operation after a certain feature map, and the Fc layer outputs the scene confidence (probability) of each scene. 1000 represents the number of candidate image scenes.

[0079]

Table 1

[0080]

[0081]

Table 2

[0082] The name of the layer Output size layer Pool_cr 1x2048 Max Pooling Fc 1x1000 Fully connected

[0083] The target scene recognition model can be pre-trained using a preset data set to determine the initial parameters of the target scene recognition model. For example, when the target scene recognition model is based on Resnet101, and based on the deep neural network model of the structure in Table 1, in order to converge quickly and ensure a good recognition effect, the open source ResNet101 model parameters pre-trained on the ImageNet data set can be used as the initial model parameters of the target scene recognition model, that is, Conv1-Conv5 uses the parameters of the ResNet101 pre-trained on the ImageNet data set. In addition, since the target scene recognition model is trained again using the above-mentioned training sample set obtained in step S230, the model has only two situations (including the target scene and not including the target scene) for the classification output of a video image, which is actually a binary classification problem, so after determining the initial parameters, the Fc layer described in Table 2 needs to be replaced, and replaced with a classification layer Fc_cr (the number of candidate scenes is 2), and for this layer Fc_cr, a Gaussian distribution with a variance of 0.01 and a mean of 0 can be used for initialization. The ImageNet dataset is a large-scale open source dataset for general object recognition. The initial model parameters can also be referred to as the parameters of the Imagenet pre-trained model, which refers to the parameter weights of the model obtained by training a deep neural network model based on the ImageNet dataset. Of course, other existing open source datasets (e.g., open-image) and / or other structured models can also be used, as long as the model can be used for scene recognition after training.

[0084] More details of step S240 will be described later in conjunction with Figure 7 Describe in detail.

[0085] By reference Figure 2 The described training method for the target scene recognition model can screen out a video library with identification information associated with the target scene from the initial video library based on the keywords of the target scene, and then select a training sample set from the video image library based on the distribution characteristics of multiple video images in the video image library of the video library and a small number of prototype images containing the target scene, and then use the training sample set to train the target scene recognition model, so that irrelevant video images can be effectively eliminated, thereby greatly reducing the samples to be labeled. In addition, when the target scene to be recognized is changed, only the prototype image needs to be replaced and the keywords of the video library need to be screened. That is, this method of generating training samples for training the target scene recognition model can be directly migrated to other target scene recognition tasks.

[0086] Figure 4A-4B More details in step S230 are shown.

[0087] Optionally, the distribution characteristics of the multiple video images involved in this step are obtained by clustering the multiple video images, that is, dividing the multiple video images into multiple groups, and making the video images in each group as similar as possible to the video images in other groups.

[0088] Optionally, each video image has feature information and can be represented by a feature vector (represented by embedding). The clustering of the multiple video images can be clustering the feature vectors of the multiple video images according to a clustering algorithm, for example, through a K-means algorithm, a mean shift clustering algorithm, a density-based clustering method, etc. This article takes the K-means algorithm as an example.

[0089] First, if Figure 4A As shown, in step S2301, the multiple video images are clustered to obtain N1 clusters, where N1 is an integer greater than 1 and less than the number of videos in the video library.

[0090] That is, the number of cluster centers is also N1. Optionally, N1 is set to be smaller than the number of videos in the video library to avoid image samples in each cluster being video images of the same video, for example, the value of N1 is 1 / 10 of the number of videos.

[0091] Optionally, the feature vector of each video image (and the feature vector of the prototype image required for the next step) can be obtained in the following manner: a deep neural network model (e.g., a ResNet101 model) pre-trained on a preset data set (e.g., an ImageNet data set) as described in the previous text in combination with Tables 1 and 2 can be used (i.e., Conv1-Conv5 uses the parameters of ResNet101 pre-trained on the ImageNet data set), and the fully connected layer is removed from the output layer structure. After each video image (and prototype image) is input into the pre-trained ResNet101 model, the one-dimensional feature vector output from the maximum pooling layer in the output layer structure is used as the feature vector of the video image (and prototype image).

[0092] In addition, the initial parameters of the deep neural network model may be variable. For example, when the deep neural network model is also used as a target scene recognition model, the parameters of the target scene recognition model will be continuously updated and adjusted during the training process. When a new training sample set needs to be mined, the feature vectors (embedding) of these images can be generated based on the target scene recognition model with updated or adjusted parameters (without the fully connected layer).

[0093] Of course, other methods can also be used to obtain the feature vector of each video image (and prototype image), as long as the feature information of these images can be expressed as feature vectors. For example, other existing open source datasets (e.g., open-image) and / or other structured models (e.g., other large-scale classification models or image metric learning models) can be used to similarly obtain the feature vector of the image; the feature vector of the image can also be obtained by obtaining traditional image features based on SIFT, color histogram, HOG, etc.

[0094] Then, in step S2302, the positive sample set and the negative sample set are obtained based on the distance between each prototype image and the cluster centers of the N1 clusters, and the distance between the cluster centers of the N1 clusters.

[0095] This step can be considered as mining positive and negative samples from the multiple video images based on the prototype images according to the above distances, and the positive and negative samples are obtained by diffusion of the prototype images.

[0096] Further references Figure 4B .like Figure 4B As shown, in step S2302-1, a first-level positive sample set is obtained based on the distance between each prototype image and the cluster center of the N1 clusters.

[0097] Optionally, the distance between an image and a cluster center can also be understood as the distance between the feature vector of the image and the cluster center (a vector with the same dimension as the feature vector). For the feature vector of each prototype image, the N2 cluster centers that are closest (most similar) to the feature vector of the prototype image among the N1 cluster centers are used as the first-level cluster centers, where N2 is an integer greater than 1; and all image samples in the clusters to which all the first-level cluster centers corresponding to the N prototype images belong are used as the first-level positive samples included in the first-level positive sample set.

[0098] Optionally, N2 may be an empirical value, for example, N2 may be 10. For example, there are 1000 cluster centers in total. For each prototype image, the 10 cluster centers that are closest to the feature vector of the prototype image among the 1000 cluster centers are used as the first-level cluster centers of the prototype image. Since the prototype image definitely contains the target scene, the image samples in the clusters of these first-level cluster centers are also likely to contain the target scene. For example, at least 50% of the image samples in the clusters to which these first-level cluster centers belong contain the target scene.

[0099] However, since the clustering process is not completely accurate, there may be one or some other cluster centers that are not the primary cluster centers, and at least a portion of the image samples in the clusters to which they belong also contain the target scene, so image samples in other clusters may be appropriately considered. The samples in the cluster to which the primary cluster center in step S2302-1 belongs (also referred to as the primary cluster) are primary positive samples, and at least a portion of the image samples in the cluster to which the secondary cluster center obtained in step S2302-2 belongs (also referred to as the secondary cluster) are secondary positive samples. In addition, the training sample set should also include negative samples.

[0100] Therefore, in step S2302-2, the secondary positive sample set and the negative sample set are obtained based on the distances between the cluster centers of the N1 clusters, and in step S2302-3, the primary positive sample set and the secondary positive sample set are used as the positive sample set.

[0101] Optionally, obtaining the second-level positive samples in the second-level positive sample set may include the following operations: Perform the following operations i-ii for each first-level cluster center.

[0102] First, in operation i, at least one cluster center among the cluster centers of the N1 clusters that is not a primary cluster center and is closest to the primary cluster center is determined as a secondary cluster center corresponding to the primary cluster center.

[0103] Specifically, the image samples included in the clusters to which the non-first-level cluster centers that are closer to the first-level cluster centers belong are more likely to contain the target scene. Therefore, generally, only one or two cluster centers that are closest to the first-level cluster centers can be selected as the second-level cluster centers corresponding to the first-level cluster centers, because only a few or no image samples in the remaining clusters may include the target scene, so they can be ignored. Of course, if the computing power of the processing device is sufficient, more cluster centers can also be selected as second-level cluster centers to obtain more second-level positive samples containing the target scene.

[0104] Then, in operation ii, the distance between the first-level cluster center and the corresponding second-level cluster center is determined, and the determined distance is used as the first distance; and in operation iii, the image samples that meet the first condition in the cluster to which the corresponding second-level cluster center belongs are determined as the second-level positive samples used to constitute the second-level positive sample set, wherein the distance between the image samples that meet the first condition and the first-level cluster center is the second distance, and the second distance and the first distance satisfy a first preset threshold condition.

[0105] For example, the first preset threshold condition satisfied by the second distance (expressed as d2) and the first distance (expressed as d1) may be that a first ratio of d2 to d1 is less than or equal to r1, and r1 is greater than 0 and less than or equal to 1.

[0106] More specifically, assuming that the above r1 is 3 / 4, for the first-level cluster center k1, its corresponding second-level cluster center is k1' (the cluster center closest to k1), and the distance (first distance) between the second-level cluster center k1' and the first-level cluster center k1 is d1', and the ratio of the distance (second distance) d2' and d2" between the feature vectors of the image samples s1' and s2' of the cluster to which the second-level cluster center k1 belongs and the first-level cluster center k1 to d1' is 1 / 2, therefore, the image samples s1' and s2' can be used as second-level positive samples. Alternatively, for the case where there are two second-level cluster centers, for the first-level cluster center k1, its corresponding second-level cluster centers are k11 and k12 (the two cluster centers closest to k1). Cluster center), and the distance (first distance) between the secondary cluster center k11 and the primary cluster center k1 is d11, and the distance (first distance) between the secondary cluster center k12 and the primary cluster center k1 is d12. For the secondary cluster center k11, the ratio of the second distance d21 and d22 of the feature vectors of the image samples s1 and s2 of the cluster to which it belongs to and the primary cluster center k1 to d11 is 1 / 2 (less than 3 / 4); for the secondary cluster center k11, the ratio of the second distance d23 of the feature vector of the image sample s3 of the cluster to which it belongs and the primary cluster center k2 to d12 is 3 / 4, therefore, the image samples s1-s3 in the clusters to which the secondary cluster centers k11 and k12 belong can be used as secondary positive samples.

[0107] The above operations of S2302-1 and S2302-2 specifically describe the process of mining positive samples (including primary positive samples and secondary negative samples) from the video image library based on the distribution characteristics of the prototype image and the video image in the video image library in the embodiment of the present disclosure. The process of mining negative samples is described below. Similarly, operations iv-vi can also be performed for each primary clustering center.

[0108] Continue to refer Figure 4B In operation iv, the cluster centers of N3 clusters that are farthest from the primary cluster center among the cluster centers of the N1 clusters are determined, and the determined cluster centers of the N3 clusters are used as the N3 candidate negative sample cluster centers corresponding to the primary cluster center, where N3 is an integer greater than 1.

[0109] Optionally, the N3 candidate negative sample cluster centers may be sorted in descending order according to their distances from the primary cluster center, so as to simplify the process of determining the negative sample cluster center corresponding to each primary cluster center in the next step.

[0110] In operation v, based on the distances between each of the N3 candidate negative sample cluster centers and the primary cluster center, and the distances between each of the N3 candidate negative sample cluster centers and the corresponding secondary cluster center, one of the N3 candidate negative sample cluster centers is selected as the negative sample cluster center corresponding to the primary cluster center.

[0111] Optionally, this step may specifically include performing the following operations for each first-level cluster center: performing the following process for each candidate negative sample cluster center corresponding to the first-level cluster center until the candidate negative sample cluster center is determined as the negative sample cluster center corresponding to the first-level cluster center: determining the distance between the candidate negative sample cluster center and the first-level cluster center, and using the determined distance as the third distance; using the second-level cluster center that is closest to the candidate negative sample cluster center among all second-level cluster centers as the second-level cluster center corresponding to the candidate negative sample cluster center; determining the distance between the candidate negative sample cluster center and the corresponding second-level cluster center, and using the determined distance as the fourth distance; and when the third distance and the fourth distance meet the second preset threshold condition, determining the candidate negative sample cluster center as the negative sample cluster center corresponding to the first-level cluster center.

[0112] For example, the second threshold condition may be a fourth distance (eg, Figure 5The second ratio of the first distance (ds2) to the third distance (for example, ds1) is greater than r2, r2 is greater than 0 and less than or equal to 1, for example, r2 = 0.5. That is to say, the distance between the negative sample cluster center corresponding to the first-level cluster center and all the second-level cluster centers should be far enough. The consideration for this setting is: since only a small number of samples in the cluster to which the second-level cluster center belongs may contain the target scene, and the second-level cluster center is relatively close to the first-level cluster center, if the distance between the candidate negative sample cluster center and all the second-level cluster centers is still far enough (judged based on the distance between the candidate negative sample cluster center and its nearest second-level cluster center), then it means that the samples in the cluster to which the candidate negative sample cluster center belongs are less likely to contain the target scene.

[0113] Since it is necessary to determine for each candidate negative sample cluster center whether the candidate negative sample cluster center can be used as the negative sample cluster center of the first-level cluster center, and after the negative sample cluster center is determined, the subsequent candidate negative sample cluster center corresponding to the first-level cluster center will no longer be judged, and the farther the distance between a certain cluster center and the first-level cluster center is, the greater the possibility that the cluster center is the negative sample cluster center, therefore, the N3 candidate negative sample cluster centers can be sorted in descending order according to the distance from the first-level cluster center, so that the candidate negative sample cluster center that is more likely to be the negative sample cluster center can be placed in the front position, so that the negative sample cluster center corresponding to the first-level cluster center can be found more quickly.

[0114] In addition, for each first-level cluster center, there may be a situation where the negative sample cluster center corresponding to the first-level cluster center vector does not exist (the distances between other cluster centers and the first-level cluster center are not far enough). In this case, the determination of the negative sample cluster center corresponding to the first-level cluster center will be stopped, and the determination of the negative sample cluster center corresponding to the next first-level cluster center will be carried out.

[0115] Finally, in operation v, all image samples in the cluster to which the negative sample cluster center corresponding to the first-level cluster center belongs are used as negative image samples for constituting a negative sample set.

[0116] Figure 5 The above reference is exemplarily shown. Figure 4A-4B A schematic diagram of the result of the described method mining positive and negative samples from the multiple video images in the video image library.

[0117] like Figure 5 As shown, examples of four samples are shown (identified by circles with different fills).

[0118] Figure 5In the figure, the pure white fill pattern represents the first-level positive sample in the cluster to which the first-level cluster center belongs, the pure gray fill pattern represents multiple samples in the cluster to which the second-level cluster center belongs (not the second-level positive sample), the vertical line fill pattern represents the second-level positive sample in the cluster to which the second-level cluster center belongs, the oblique line fill pattern represents the negative sample, and the X in the figure represents each cluster center.

[0119] exist Figure 5 In the example, the secondary cluster includes multiple samples, but some of them may be negative samples (i.e., do not contain the target scene). Therefore, a small number of samples that are more similar to the image samples in the cluster to which the primary cluster center belongs are screened out as secondary positive samples through the first preset threshold condition described above, so as to complete the cleaning of negative samples in the cluster to which the secondary cluster center belongs (i.e., the negative samples in the cluster to which the secondary cluster center belongs are not used). In addition, the negative sample cluster far away from all the secondary cluster centers can be selected through the second preset threshold condition described above, so that the accuracy of the screened negative samples can be higher.

[0120] It should be noted that the selection of the above-mentioned secondary cluster center and negative sample cluster center adopts the distance judgment between cluster centers. Of course, other methods can be used, for example, using image samples in each cluster for judgment, such as using the nearest sample distance between samples in any two clusters to replace the distance between cluster centers, and similarly performing various distance judgments, and adjusting parameters (for example, the first ratio, the second ratio, etc.) accordingly. For example, the first cluster includes A image samples, and the second cluster includes B image samples. The distance between the rth image sample in the first cluster (r=1, 2, ...A, one of them) and the sth image sample in the second cluster (s=1, 2, ...B, one of them) is the nearest distance between other samples in the first cluster and the second cluster. Therefore, this distance is regarded as the sample distance between the two clusters and is used to determine the secondary cluster center and the negative sample cluster center.

[0121] The above process of obtaining positive and negative sample sets can be simply summarized as follows: Figure 4C The process shown in the figure, firstly, based on the distribution characteristics of the video image library (multiple video images) and the prototype image, the first-level positive sample is obtained, and then the second-level cluster center is obtained based on the first-level cluster center determined in the process of obtaining the first-level positive sample, and then at least a part of the image samples in the second-level cluster center is determined as the second-level positive sample. In addition, the acquisition of negative samples is also based on each second-level cluster center and the first-level cluster center.

[0122] Through the above reference Figures 4A-4C and Figure 5The described method can mine multiple positive samples and negative samples from a video library (video image library), and the positive samples and negative samples can constitute a training sample set to train the target scene recognition model. The target scene recognition model trained based on the training sample set can recognize the target scene to a certain extent, and this method can greatly reduce the samples to be labeled, and when the target scene to be recognized is changed, the model training method can be easily migrated directly to other target scene recognition tasks.

[0123] However, usually, the following combination Figure 6 As described above, the obtained positive and negative samples are mined through the distribution characteristics of the video images in the video image library (such as the distances between the cluster centers or the distances between the samples in the clusters), and there will be certain noise.

[0124] Figure 6 The distribution of positive samples mined based on a video library of 6 videos of the example is shown.

[0125] In the figure below, 1 to 6 represent 6 videos. The circled feature vector distribution in each video is, for example, video 1 has 4 video images, and has 4 feature vectors. Each feature vector is marked with + / - to indicate whether the video image contains the target scene. It can be seen that videos 2 and 4 contain target scenes. For this kind of situation where each video contains many video images, whether the video contains the target scene (for example, shooting) can be judged based on whether these video images are target scenes. This can be regarded as a multi-instance learning problem, where each video can be regarded as an image "package".

[0126] The gray dotted box represents a pure positive sample (such as "shooting" pictures). It is expected that the mined positive samples are as clean as possible, but in reality, they cannot be absolutely pure. For example, the black solid box represents the noisy samples (including false positive positive samples) actually mined by the above method, where the triangle represents the feature vector of the prototype image input by the user. Since the mining of positive samples is driven by the prototype image, it can be seen that the mined positive samples (including false positive positive samples) are distributed around the feature vector of the prototype image. The false positive positive samples (actually negative samples) in the black solid box have a certain correlation with the target scene (such as shooting), but are not the target scene (for example, they are actually passing the ball). In addition, there may also be false negative negative samples (actually positive samples).

[0127] Therefore, in response to the above problem, after mining the above positive and negative samples, noise samples (including false positive positive samples and false negative negative samples) can be identified in the subsequent training stage and the identified noise samples can be discarded, so that the labels of positive and negative samples used for actual model training (parameter adjustment) can be as accurate as possible, so that the recognition accuracy of the target scene recognition model trained with these positive and negative samples can be improved while reducing the amount of annotation and simplifying scene migration, and the fault tolerance of ambiguous samples can also be improved.

[0128] Therefore, for Figure 2 In step S240, you can also refer to Figure 7 To describe each sub-step.

[0129] Figure 7 Further sub-steps of training the target scene recognition model using a training sample set including noise samples are shown.

[0130] like Figure 7 As shown, in step S2401, the target scene recognition model is pre-trained using a preset data set to determine the initial parameters of the target scene recognition model and adjust the structural parameters of the output layer of the target scene recognition model.

[0131] As described above for step S240, the open-source ResNet101 model parameters pre-trained on the ImageNet data set can be used as the initial model parameters of the target scene recognition model, i.e., Conv1-Conv5 uses the parameters of the ResNet101 pre-trained on the ImageNet data set. In addition, due to the above-mentioned image sample set obtained, the target scene recognition model is trained again, and there are only two situations (including the target scene and not including the target scene) for the classification output of an image in this model, which is actually a binary classification problem, so it is necessary to replace the Fc layer described in Table 2, and replace it with a new classification layer Fc_cr (the number of candidate scenes is 2), and for this layer Fc, a Gaussian distribution with a variance of 0.01 and a mean of 0 can be used for initialization. However, all parameters of the initialized target scene recognition model are set to a state that needs to be learned (need to be adjusted), so as to train using the training sample set (including noise samples) obtained by mining, and constantly adjust each parameter of the model.

[0132] For model training, the training sample set can generally be divided into multiple batches, because the model may not be able to process all training samples at once, and then the model parameters can be adjusted once based on the calculation results of the training samples in each batch. Then, the next training sample subset is used to train the model with adjusted parameters for the next time, and parameter adjustment continues. In addition, a training sample set can be used for model training multiple times until the model parameters converge.

[0133] In step S2402, the training sample set is divided into a plurality of training sample subsets.

[0134] For example, the training sample set is generally processed in batches, each batch (batch) includes bs video images, corresponding to a training sample subset, that is, the value of batchsize is bs, and bs is an integer greater than 1. And, the model parameters are adjusted once based on the forward calculation results of each batch (batch) of video images, as will be described later.

[0135] In step S2403, the following operations are performed for each training sample subset (including bs video images).

[0136] First, operation i randomly selects a part of positive samples (positive sample subset) and a part of negative samples (negative sample subset) from the positive sample set and the negative sample set according to preset proportions, as a reference image sample set for the training sample subset (the included positive and negative sample subsets are represented by pos and neg, respectively).

[0137] For example, if the training sample set includes a total of 50,000 video images (for example, 30,000 positive sample sets and 20,000 negative sample sets), and the preset ratio is 1 / 5, then 1 / 5 of the video images in the positive sample set and the negative sample set are selected respectively (6,000 positive samples, 4,000 negative samples, and a total of 10,000 video images) as the reference image sample set of the training sample subset (bs is, for example, 100).

[0138] Here, by randomly selecting a part of positive and negative samples, the processing complexity can be reduced, and different reference image sample sets can be randomly selected for different training sample subsets, so that the regularization effect can be achieved to improve the accuracy of subsequent noise sample removal.

[0139] Then, in operation ii, valid image samples of the training sample subset are determined based on the distribution characteristics and labels of the reference image samples in the reference image sample set and the labels of the image samples in the training sample subset.

[0140] More specifically, first, the distribution characteristics of the reference image samples in the reference image sample set can also be obtained by clustering, that is, the positive sample subset and the negative sample subset can be clustered separately to obtain N4 positive sample clusters and N4 negative sample clusters, where N4 is a positive integer greater than 1.

[0141] For example, N4=20, the positive sample subset and the negative sample subset (represented by pos and neg, respectively) are clustered to obtain 20 positive sample clusters and 20 negative sample clusters. Therefore, there are 20 cluster centers for the positive sample subsets (called positive cluster centers) and 20 cluster centers for the negative sample subsets (called negative cluster centers), and each cluster may only include video images of positive samples labeled "1", only include video images of negative samples labeled "0", and include video images of both positive samples labeled "1" and negative samples labeled "0".

[0142] Then, determine the reference image sample in the positive sample subset in operation i that is closest to each positive cluster center of the N4 positive sample clusters, and determine the reference image sample in the negative sample subset in operation i that is closest to each negative cluster center of the N4 negative sample clusters, to obtain 2*N4 reference image samples.

[0143] For example, when N4=20, 40 reference image samples can be obtained, and the label of each reference image sample is known.

[0144] Next, for each image sample among the bs image samples in the training sample subset, determine a reference image sample among the 2*N4 reference image samples that is closest to the image sample, and when the label of the reference image sample that is closest to the image sample is the same as the label of the image sample, determine that the image sample is a valid image sample.

[0145] That is to say, for example, for the p-th image sample among bs image samples (p is greater than or equal to 1 and less than or equal to bs), the distance between the feature vector of the p-th image sample and the feature vector of each reference image sample in 2*N4 reference image samples is calculated, and the distance to the feature vector of a certain reference image sample is determined to be the closest, then it is further determined whether the label (1 or 0) of the p-th image sample is consistent with the label of the reference image sample. If so, the p-th image sample is determined to be a valid image sample that can be used for training. Otherwise, it means that the accuracy of the label of the p-th image sample may be ambiguous, and it is very likely to be a noise sample, and therefore should be discarded.

[0146] Through this step, the noise samples in the bs image samples can be removed, that is, these noise samples will not be used for parameter adjustment of the target scene recognition model, and the remaining image samples will be used for parameter adjustment of the target scene recognition model.

[0147] In addition, in operation ii, for bs image samples in the training sample subset, feature vectors of bs image samples in the training sample subset and feature vectors of reference image samples in the reference image sample set are used for distance calculation and clustering. Therefore, before the clustering process in operation ii, the image samples of the training sample subset and all reference image samples (including noise samples) in the reference image sample set randomly selected for the training sample subset can be respectively input into the target scene recognition model, and the target scene recognition model outputs feature vectors of each image sample in the maximum pooling layer based on the current model parameters.

[0148] Obviously, after each of the bs image samples in the training sample subset is input into the target scene recognition model, a prediction probability vector (two-dimensional feature vector corresponding to the binary classification) can also be output in the fully connected layer (Fc_cr), for example (0.95, 0.05), which means that the probability that the image sample is a positive sample (labeled as 1) is 0.95, and the probability that it is a negative sample (labeled as 0) is 0.05. If the image sample is determined to be a valid image sample according to the above operation, the prediction probability vector will be used to adjust the parameters of the target scene recognition model. Usually, after the valid image samples in the training sample subset are forward calculated to obtain the error value, the parameters of the target scene recognition model are adjusted once.

[0149] Therefore, finally, in step S2403, the current parameters of the target scene recognition model are adjusted using the valid image samples in the training sample subset.

[0150] Optionally, the error back propagation algorithm can be used in combination with the gradient descent method to perform parameter adjustment.

[0151] Specifically, after determining the valid image samples in the training sample subset that can be used to train the target scene recognition model, the following cross entropy loss function is used to calculate the loss of the forward calculation of each valid image sample:

[0152]

[0153] Where y=1 represents a positive label and 0 represents a negative label. represents the predicted positive label probability, Represents the predicted negative label probability. The losses corresponding to each valid image sample are added together as the total loss of the training sample subset, which is used for backpropagation to the target scene recognition model (such as a convolutional neural network model), to calculate the gradient and update the parameters of the target scene recognition model (such as convolutional template parameters and compilation parameters, etc.), thereby achieving a parameter adjustment (weight optimization).

[0154] After adjusting the parameters of the target scene recognition model for the training sample subset, the same operation is continued for the next training sample subset. In addition, after the parameters of the target scene recognition model are adjusted for all batches of image samples (training sample subsets) in the training sample set, that is, after an epoch, the training sample set is reused (the noise samples determined in the previous training process may or may not be discarded) to continue adjusting the parameters of the target scene recognition model in batches until the parameters of the target scene recognition model converge (for example, the target loss function converges).

[0155] Optionally, after a preset number of parameter adjustments are made based on the current training sample set, or the parameters of the target scene recognition model based on the current training sample set have converged, a new round of positive and negative sample mining can be performed using the target scene recognition model (e.g., Conv1-Conv5 layers and the maximum pooling layer) based on the current parameters of the target scene recognition model (as described in the previous reference). Figure 2-5 As described above), an updated training sample set is obtained, and then the following is performed: Figure 7 The training process described above continues to train the target scene recognition model.

[0156] It can be seen that by referring to Figure 7 The training process described can identify noise samples (including false positive samples and false negative samples) and discard the identified noise samples, so that the labels of positive and negative samples used for actual model training (parameter adjustment) are as accurate as possible, so that the recognition accuracy of the target scene recognition model trained with these positive and negative samples can be improved while reducing the amount of annotation and simplifying scene migration, and the fault tolerance of ambiguous samples can also be improved.

[0157] According to another aspect of the present disclosure, a method for identifying a target scene is also provided. Figures 8A-8B The flowchart of the method for identifying a target scene according to an embodiment of the present disclosure is shown. The training method can be executed by any computing device with processing function, such as Figure 1B The server (including cloud server) and / or user terminal shown in, or Figure 1D The computing devices at each node in the blockchain system shown are executed. Fig. 8AAs shown, in step S810, a video to be identified is obtained.

[0158] For example, the method can be executed in a server, and the video to be identified can be stored locally in the server, or the user terminal can send the video to be identified or its download address to the server. Of course, the method can be executed in any device with processing function, and the video to be identified can be obtained from the outside or called locally.

[0159] In step S820, a to-be-recognized video image library is generated based on the to-be-recognized video.

[0160] For example, a library of video images to be identified is generated by extracting frames from the video to be identified.

[0161] In step S830, a target scene recognition model is used to identify whether the video image in the video image library to be identified contains a target scene, wherein the target scene recognition model is a reference Figure 2-7 The training is carried out using the training method described.

[0162] More details of step S830 have been described in detail in the previous training method, so they will not be repeated here.

[0163] In step S840, when at least one video image in the to-be-recognized video image library includes the target scene, it is determined that the to-be-recognized video includes the target scene.

[0164] In addition, a manual verification step can be added to the actual application of the trained target scene recognition model (actually used for target scene recognition), and more prototype images can be obtained based on the results of the manual verification, and corrections can be made for erroneous cases, thereby further optimizing the model.

[0165] Therefore, the target scene recognition method may further include the following steps.

[0166] like Figure 8B As shown, in step S850, the target scene recognition model is used to perform target scene recognition on multiple videos to be recognized, and multiple recognition results are obtained.

[0167] For example, in the actual application of the target scene recognition model, each recognition result is saved.

[0168] In step S860, a review result of at least one of the multiple videos to be identified is received.

[0169] Optionally, at least one of the multiple videos to be identified is manually reviewed to obtain at least one manual review result. Generally, a spot check is performed on the videos to be identified whose recognition results are that they do not contain the target scene, because if the video to be identified actually contains the target scene, a new prototype image can be obtained from it, making full use of existing resources.

[0170] In step S870, when the recognition result for any video to be recognized indicates that it does not contain the target scene but the manual review result indicates that it contains the target scene, the video image containing the target scene in the image library of the video to be recognized is used as a new prototype image for selecting a new training sample set for the target scene recognition model.

[0171] Optionally, when it is manually determined that a certain video to be identified as not containing the target scene by the model actually contains the target scene, the erroneous image sample is manually located (for example, determining which frame of the video image contains the target scene) to extract the image sample and store it as a new prototype image. Therefore, when the model is trained next time, the training sample set can be mined based on the updated prototype image and used for the next training of the model, as shown in FIG. Figure 2-5 Just as described.

[0172] Optionally, when it is manually determined that a certain video to be identified by the model as containing the target scene does contain the target scene, a video frame (video image) containing the target scene can also be located from the video to be identified and used as a new prototype image.

[0173] Optionally, when it is manually determined that a video to be identified as containing a target scene by the model does not actually contain the target scene, it is impossible to add a prototype image when considering this situation, and a small recognition error can be allowed, so the embodiments of the present disclosure may not consider this situation.

[0174] The example process can be referred to Fig. 9 . Fig. 9 The diagram shows an interaction diagram involved in the target scene recognition method of the present disclosure with a manual verification step added.

[0175] like Fig. 9As shown, the user inputs multiple videos, and the trained target scene recognition model will output multiple recognition results (not shown). Some videos can be randomly selected for manual inspection. If the manual inspection finds that the video judged as no (does not contain the target scene) is actually a shot, the manual error sample feedback is performed and which frame image sample in the video contains the target scene, that is, the positive sample, and then the image sample is added as a new prototype image to the prototype image, and the sample mining of the training sample set for the next training is performed for the next model training. The updated target scene recognition model can be used for new target scene recognition tasks.

[0176] Based on reference Figures 8A-8B and 9 described target scene recognition method, in addition to the above reference Figure 2-7 In addition to the advantages of the training method described, since manual review is introduced and prototype images are added based on the inconsistency between the manual review results and the recognition results (the recognition results do not contain the target scene), more prototype images can be obtained and corrections can be made for erroneous cases, so that the model can be further optimized.

[0177] According to yet another aspect of the present disclosure, a training device for a target scene recognition model is also provided.

[0178] Figures 10A-10B A structural block diagram of a training device 1000 for a target scene recognition model is shown.

[0179] like Fig. 10A As shown, the training device includes a first acquisition module 1010 , a first image library generation module 1020 , a sample selection module 1030 and a training module 1040 .

[0180] The first acquisition module 1010 is used to acquire N prototype images and a video library for model training, wherein the N prototype images contain the target scene, and the videos in the video library have identification information associated with the target scene, wherein N is an integer greater than 1.

[0181] The first image library generation module 1020 is used to obtain a video image library based on the video library, and the video image library includes multiple video images. For example, all video images constituting each video can be obtained by video frame extraction.

[0182] The sample selection module 1030 is used to select a training sample set for a target scene recognition model from the multiple video images based on the N prototype images and the distribution characteristics of the multiple video images, wherein the training sample set includes a positive sample set containing the target scene and a negative sample set not containing the target scene.

[0183] The training module 1040 is used to train the target scene recognition model using the training sample set.

[0184] More specifically, if Fig. 10B The first acquisition module 1010 may include an acquisition submodule and a screening submodule. The acquisition submodule is used to acquire an initial video library, keywords of a target scene, and a plurality of prototype images, and the screening submodule is used to screen out a video library having identification information associated with the target scene from the initial video library based on the content of the target scene, using keywords identical to the content of the target scene or optionally using multiple keywords with semantic extension.

[0185] The sample selection module 1030 may include a clustering submodule, a distance determination submodule, and a sample mining submodule.

[0186] The clustering submodule is used to cluster the multiple video images to obtain N1 clusters, where N1 is an integer greater than 1 and less than the number of videos in the video library.

[0187] The distance determination submodule is used to calculate the distance between each prototype image and the cluster centers of the N1 clusters, the distance between the cluster centers of the N1 clusters, and the distance between the samples in the cluster to which the secondary cluster center belongs and the corresponding primary cluster center as described below.

[0188] The sample mining submodule is used to mine positive and negative sample sets (training sample sets) from the multiple video images based on the distances determined by the distance determination submodule.

[0189] According to an embodiment of the present disclosure, the sample mining submodule is configured to perform the following operations.

[0190] First, the sample mining submodule obtains a first-level positive sample set based on the distance between each prototype image and the cluster centers of the N1 clusters.

[0191] Specifically, for each prototype image, the cluster centers of the N2 clusters that are closest to the prototype image among the cluster centers of the N1 clusters are taken as the first-level cluster centers corresponding to the prototype image, where N2 is an integer greater than 1; and all image samples in the clusters to which all the first-level cluster centers corresponding to the N prototype images belong are taken as the first-level positive samples included in the first-level positive sample set.

[0192] Secondly, the sample mining submodule obtains a secondary positive sample set and the negative sample set based on the distances between the cluster centers of the N1 clusters.

[0193] Specifically, for the secondary positive sample set, the sample mining submodule determines, for each first-level cluster center, at least one cluster center among the cluster centers of the N1 clusters that is not a first-level cluster center and is closest to the first-level cluster center, as the second-level cluster center corresponding to the first-level cluster center, determines the distance between the first-level cluster center and the second-level cluster center, and uses the determined distance as the first distance; and determines the image samples that meet the first condition in the cluster to which the second-level cluster center belongs as the second-level positive samples included in the secondary positive sample set, wherein the distance between the image samples that meet the first condition and the first-level cluster center is the second distance, and the second distance and the first distance satisfy a first preset threshold condition.

[0194] Then, for the negative sample set, the sample mining submodule performs the following operations: for each first-level cluster center, determine the cluster centers of the N3 clusters that are farthest from the first-level cluster center among the cluster centers of the N1 clusters, and use the determined cluster centers of the N3 clusters as the N3 candidate negative sample cluster centers corresponding to the first-level cluster center, where N3 is an integer greater than 1; based on the distances of the N3 candidate negative sample cluster centers from each of the first-level cluster centers and the distances from each of the corresponding second-level cluster centers, select one of the N3 candidate negative sample cluster centers as the negative sample cluster center corresponding to the first-level cluster center; and use all image samples in the cluster to which the negative sample cluster center corresponding to the first-level cluster center belongs as the negative samples included in the negative sample set.

[0195] Specifically, regarding the operation of selecting one of the N3 candidate negative sample cluster centers as the negative sample cluster center corresponding to the first-level cluster center, for each first-level cluster center, the sample mining submodule and the distance determination submodule perform the following process for each candidate negative sample cluster center corresponding to the first-level cluster center until the candidate negative sample cluster center is determined as the negative sample cluster center corresponding to the first-level cluster center: the distance determination submodule determines the distance between the candidate negative sample cluster center and the first-level cluster center, and uses the determined distance as the third distance, and determines the distance between the candidate negative sample cluster center and its corresponding second-level cluster center, and uses the determined distance as the fourth distance, and the sample mining submodule uses the second-level cluster center that is closest to the candidate negative sample cluster center among all the second-level cluster centers as the second-level cluster center corresponding to the candidate negative sample cluster center, and when the third distance and the fourth distance meet the second preset threshold condition, determines the candidate negative sample cluster center as the negative sample cluster center corresponding to the first-level cluster center.

[0196] The above operations of the various submodules included in the sample selection module 1030 can generate a training image set, but the training image set will include some noise samples (for example, false positive samples and false negative samples), so these noise samples can be discarded as much as possible in the operation of the training module.

[0197] More details of the above operations of the sample selection module 1030 are as described in the previous reference. Figure 4A-5 The content of step S230 described is basically the same, so it will not be repeated here.

[0198] The training module 1040 may include a pre-training submodule, a partitioning submodule, and a training submodule.

[0199] The pre-training submodule is used to pre-train the target scene recognition model using a preset data set to determine the initial parameters of the target scene recognition model and adjust the output structure parameters of the target scene recognition model. The preset data set can be any open source data set for classification tasks, and the target scene recognition model can be a deep neural network model.

[0200] The division submodule is used to divide the training sample set into multiple training sample subsets.

[0201] The training submodule is used to perform the following operations on each training sample subset.

[0202] First, a positive sample subset and a negative sample subset are randomly selected from the positive sample set and the negative sample set according to preset proportions, respectively, as reference image sample sets for the training sample subset.

[0203] Then, valid image samples of the training sample subset are determined based on the distribution characteristics and labels of the reference image samples in the reference image sample set and the labels of the image samples in the training sample subset.

[0204] Specifically, the positive sample subset and the negative sample subset are clustered respectively to obtain N4 positive sample clusters and N4 negative sample clusters, where N4 is a positive integer greater than 1; the reference image samples in the positive sample subset that are closest to the center of each positive cluster of the N4 positive sample clusters are determined, and the reference image samples in the negative sample subset that are closest to the center of each negative cluster of the N4 negative sample clusters are determined to obtain 2*N4 reference image samples; for each image sample in the training sample subset, a reference image sample in the 2*N4 reference image samples that is closest to the image sample is determined, and when the label of the reference image sample that is closest to the image sample is the same as the label of the image sample, the image sample is determined to be a valid image sample.

[0205] Finally, the current parameters of the target scene recognition model are adjusted using the valid image samples in the training sample subset.

[0206] More details of the above operations of the training module 1040 are as described in the previous reference. Figure 7 The content of step S240 described is basically the same, so it will not be repeated here.

[0207] Through the above reference Figures 10A-10B The training device described can mine multiple positive samples and negative samples from a video library (video image library) based on a prototype image, and the positive samples and negative samples can constitute a training sample set to train a target scene recognition model. The target scene recognition model trained based on the training sample set can recognize the target scene to a certain extent, and this method can greatly reduce the samples to be labeled, and when the target scene to be recognized is changed, the model training method can be easily migrated directly to other target scene recognition tasks.

[0208] In addition, noise samples (including false positive samples and false negative samples) can be identified and discarded, so that the labels of positive and negative samples used for actual model training (parameter adjustment) can be as accurate as possible, so that the recognition accuracy of the target scene recognition model trained with these positive and negative samples can be improved while reducing the amount of annotation and simplifying scene migration, and the fault tolerance of ambiguous samples can also be improved.

[0209] According to yet another aspect of the present disclosure, a target scene recognition device is provided.

[0210] Fig.11 A structural block diagram of a target scene recognition device 1100 is shown.

[0211] like Fig.11 As shown, the target scene recognition device 1100 includes a second acquisition module 1110 , an image generation module 1120 , a recognition module 1130 and a determination module 1140 .

[0212] The second acquisition module 1110 is used to acquire the video to be identified.

[0213] The second image library generating module 1120 is used to generate a to-be-recognized video image library based on the to-be-recognized video.

[0214] The recognition module 1130 is used to identify whether the video image in the video image library to be identified contains a target scene through a target scene recognition model, wherein the target scene recognition model can be local or obtained from the outside, and the target scene recognition model is based on the reference Figure 2-7 The training is obtained by training using the training method described.

[0215] The determination module 1140 is configured to determine that the video to be identified includes the target scene when at least one video image in the video image library to be identified includes the target scene.

[0216] Optionally, the target scene recognition device may also include a manual review module, which is used to receive a review result of at least one of the multiple videos to be recognized, and when the recognition result of any video to be recognized indicates that it does not contain the target scene but the review result indicates that it contains the target scene, the video image containing the target scene in the image library of the video to be recognized is used as a new prototype image for selecting a new training sample set for the target scene recognition model. Optionally, the new prototype image is saved locally, or the new prototype image is sent to a location where other prototype images are stored.

[0217] Optionally, the target scene recognition device 1100 may include a reference Fig. 10A A training device 1000 for a target scene recognition model is described.

[0218] More details on each module are in the previous reference Figure 8A-9 The descriptions are basically the same, so they will not be repeated here.

[0219] Through the target scene recognition device of the embodiment of the present disclosure, in addition to the above reference Figure 2-7 In addition to the advantages of the training method described, since manual review is introduced and prototype images are added based on the inconsistency between the manual review results and the recognition results (the recognition results do not contain the target scene), more prototype images can be obtained and corrections can be made for erroneous cases, so that the model can be further optimized.

[0220] It should be noted that in reference Figure 10A-11 In the described devices, each device is described as including different modules according to the division of each function, but the device may include more or fewer modules according to different functional division methods.

[0221] According to yet another aspect of the present disclosure, a computing device is provided. Fig.12 The structure block diagram of the computing device 1200 according to the embodiment of the present disclosure is shown. The computing device includes: a processor; and a memory having instructions stored thereon, and when the instructions are executed by the processor, the processor executes the following steps: Figure 2-7 The various steps of the training method described and reference Figure 8A-9 The various steps of the target scene recognition method.

[0222] The computing device may be a computer terminal, a mobile terminal or other device with computing and processing capabilities. The computing device may also participate in constituting or include the training apparatus provided in the embodiment of the present disclosure.

[0223] The computing device may be, for example, Figure 1B The server (including cloud server) and / or user terminal shown in, or Figure 1D Computing devices at various nodes in the blockchain system shown.

[0224] The processor can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X84 architecture or an ARM architecture.

[0225] The memory may be a non-volatile memory such as a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. It should be noted that the memory of the methods described in the present disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

[0226] The display screen of the computing device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computing device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the terminal shell, or an external keyboard, touchpad or mouse.

[0227] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the Figure 2-7 The various steps of the training method described and reference Figure 8A-9 The various steps of the target scene recognition method.

[0228] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the Figure 2-7 The various steps of the training method described and reference Figure 8A-9 The various steps of the target scene recognition method. The computer program can be stored in a computer-readable storage medium.

[0229] The storage medium mentioned above may be a non-volatile storage medium such as a read-only memory, a magnetic disk or an optical disk.

[0230] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes at least one executable instruction for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0231] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0232] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A training method for a target scene recognition model, comprising: Obtain N prototype images and a video library for model training, wherein the N prototype images contain a target scene, and the videos in the video library have identification information associated with the target scene, wherein N is an integer greater than 1; Obtaining a video image library based on the video library, wherein the video image library includes a plurality of video images; Clustering the multiple video images to obtain N1 clusters, where N1 is an integer greater than 1 and less than the number of videos in the video library; Based on the distance between each prototype image and the cluster centers of the N1 clusters, and the distance between the cluster centers of the N1 clusters, a training sample set for a target scene recognition model is selected from the multiple video images, wherein a first-level positive sample set is obtained based on the distance between each prototype image and the cluster centers of the N1 clusters; a second-level positive sample set and a negative sample set are obtained based on the distance between the cluster centers of the N1 clusters; and the first-level positive sample set and the second-level positive sample set are used as positive sample sets, and the training sample set includes the positive sample set containing the target scene and the negative sample set not containing the target scene; and The target scene recognition model is trained using the training sample set.

2. The training method according to claim 1, wherein: The identification information is at least one of a title, a description, and a tag. The videos in the video library are obtained by screening an initial video library based on the content of the target scene and the identification information.

3. The training method according to claim 1, wherein: A first-level positive sample set is obtained based on the distance between each prototype image and the cluster center of the N1 clusters, including: For each prototype image, the cluster centers of the N2 clusters that are closest to the prototype image among the cluster centers of the N1 clusters are used as the first-level cluster centers corresponding to the prototype image, where N2 is an integer greater than 1; and All image samples in the clusters to which all the first-level cluster centers corresponding to the N prototype images belong are used as first-level positive samples for constituting the first-level positive sample set.

4. The training method according to claim 3, wherein: A second-level positive sample set is obtained based on the distances between the cluster centers of the N1 clusters, including: for each first-level cluster center, Determine at least one cluster center among the cluster centers of the N1 clusters that is not a primary cluster center and is closest to the primary cluster center, and use the at least one closest cluster center as a secondary cluster center corresponding to the primary cluster center, Determining a distance between the first-level cluster center and its corresponding second-level cluster center, and using the determined distance between the first-level cluster center and the corresponding second-level cluster center as a first distance; and An image sample that meets a first condition in the cluster to which the secondary cluster center corresponding to the first-level cluster center belongs is determined as a secondary positive sample for constituting the secondary positive sample set, wherein the distance between the image sample that meets the first condition and the first-level cluster center is a second distance, and the second distance and the first distance meet a first preset threshold condition, and the first preset threshold condition is used to screen out samples that are similar to the image sample in the cluster to which the first-level cluster center belongs as the secondary positive sample.

5. The training method according to claim 4, wherein: The negative sample set is obtained based on the distances between the cluster centers of the N1 clusters, including: for each first-level cluster center, Determine the cluster centers of N3 clusters that are farthest from the primary cluster center among the cluster centers of the N1 clusters, and use the determined cluster centers of the N3 clusters as N3 candidate negative sample cluster centers corresponding to the primary cluster center, where N3 is an integer greater than 1; Based on the distances between each of the N3 candidate negative sample cluster centers and the primary cluster center, and the distances between the secondary cluster centers corresponding to the candidate negative sample cluster centers, one of the N3 candidate negative sample cluster centers is selected as the negative sample cluster center corresponding to the primary cluster center, wherein the secondary cluster center corresponding to the candidate negative sample cluster center is: the secondary cluster center that is closest to the candidate negative sample cluster center among all the secondary cluster centers corresponding to the primary cluster center; All image samples in the cluster to which the negative sample cluster center corresponding to the first-level cluster center belongs are used as negative samples for constituting the negative sample set.

6. The training method according to claim 5, wherein: Based on the distance between each of the N3 candidate negative sample cluster centers and the primary cluster center, and the distance between the N3 candidate negative sample cluster centers and the secondary cluster center corresponding to the candidate negative sample cluster center, selecting one of the N3 candidate negative sample cluster centers as the negative sample cluster center corresponding to the primary cluster center, comprising: The following process is performed for each candidate negative sample cluster center corresponding to the first-level cluster center until the candidate negative sample cluster center is determined to be the negative sample cluster center corresponding to the first-level cluster center: Determine the distance between the candidate negative sample cluster center and the primary cluster center, and use the determined distance between the candidate negative sample cluster center and the primary cluster center as the third distance; Determine the distance between the candidate negative sample cluster center and its corresponding secondary cluster center, and use the determined distance between the candidate negative sample cluster center and its corresponding secondary cluster center as a fourth distance; When the third distance and the fourth distance satisfy the second preset threshold condition, the candidate negative sample cluster center is determined as the negative sample cluster center corresponding to the first-level cluster center, and the second preset threshold condition is used to screen out negative sample clusters that are far away from all the second-level cluster centers corresponding to the first-level cluster centers.

7. The training method according to claim 1, wherein: Using the training sample set to train the target scene recognition model includes: Pre-training the target scene recognition model using a preset data set to determine initial parameters of the target scene recognition model and adjust output structure parameters of the target scene recognition model; Dividing the training sample set into a plurality of training sample subsets; For each subset of training samples: Randomly selecting a positive sample subset and a negative sample subset from the positive sample set and the negative sample set according to a preset ratio, respectively, as a reference image sample set for the training sample subset; Determining valid image samples of the training sample subset based on distribution characteristics and labels of reference image samples in the reference image sample set and labels of image samples in the training sample subset; The current parameters of the target scene recognition model are adjusted once using the valid image samples in the training sample subset.

8. The training method according to claim 7, wherein: Determining valid image samples of the training sample subset based on the distribution characteristics and labels of the reference image samples in the reference image sample set and the labels of the image samples in the training sample subset includes: Clustering the positive sample subset and the negative sample subset respectively to obtain N4 positive sample clusters and N4 negative sample clusters, where N4 is a positive integer greater than 1; Determine the reference image samples in the positive sample subset that are closest to each positive cluster center of the N4 positive sample clusters, and determine the reference image samples in the negative sample subset that are closest to each negative cluster center of the N4 negative sample clusters, to obtain 2*N4 reference image samples; For each image sample in the training sample subset, determine a reference image sample among the 2*N4 reference image samples that is closest to the image sample; if the label of the closest reference image sample is the same as the label of the image sample, determine that the image sample is a valid image sample.

9. A method for identifying a target scene, comprising: Get the video to be identified; Generate a video image library to be identified based on the video to be identified; identifying whether the video image in the to-be-identified video image library contains a target scene by using a target scene recognition model, wherein the target scene recognition model is trained by the training method according to any one of claims 1 to 8; and In the case that at least one video image in the to-be-recognized video image library includes the target scene, it is determined that the to-be-recognized video includes the target scene.

10. The method according to claim 9, further comprising: Using the target scene recognition model to perform target scene recognition on multiple videos to be recognized, and obtain multiple recognition results; receiving a review result of at least one video to be identified among the plurality of videos to be identified; When the recognition result for any video to be recognized indicates that it does not contain the target scene but its corresponding review result indicates that it contains the target scene, the video image containing the target scene in the video image library of the video to be recognized is used as a new prototype image for selecting a new training sample set for the target scene recognition model.

11. A training device for a target scene recognition model, comprising: An acquisition module, used to acquire N prototype images and a video library for model training, wherein the N prototype images contain a target scene, and the videos in the video library have identification information associated with the target scene, wherein N is an integer greater than 1; An image library generation module, used to obtain a video image library based on the video library, wherein the video image library includes a plurality of video images; A sample selection module, used for clustering the plurality of video images to obtain N1 clusters, where N1 is an integer greater than 1 and less than the number of videos in the video library; and selecting a training sample set for a target scene recognition model from the plurality of video images based on the distance between each prototype image and the cluster centers of the N1 clusters, and the distance between the cluster centers of the N1 clusters, wherein a first-level positive sample set is obtained based on the distance between each prototype image and the cluster centers of the N1 clusters; a second-level positive sample set and a negative sample set are obtained based on the distance between the cluster centers of the N1 clusters; and the first-level positive sample set and the second-level positive sample set are used as positive sample sets, and the training sample set includes the positive sample set containing the target scene and the negative sample set not containing the target scene; and A training module is used to train the target scene recognition model using the training sample set.

12. A computing device comprising: processor; and A memory having instructions stored thereon, wherein when the instructions are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 10.

Images