A scene recognition method and system

Abstract

This application proposes a scene recognition method and system. The method includes: extracting features from scene data to be recognized; and passing the extracted features in parallel through different scene networks to obtain scene recognition results for each scene network. The proposed scene recognition method and system pass the extracted features in parallel through different scene networks to obtain scene recognition results for each scene network. For example, each scene network outputs whether the scene data corresponds to a specific scene. Compared to existing technologies that only obtain the similarity between scene data and various scenes, the proposed method offers higher accuracy in scene recognition.

Description

Technical Field

[0001] This application relates to the field of recognition technology, specifically to a scene recognition method and system. Background Technology

[0002] A neural network is a mathematical model that uses a structure similar to the synaptic connections of neurons in the brain to process information. A neural network can be pre-trained and used to identify scenes. By training the neural network with samples from scene A, and then inputting the scene data into the trained neural network, the similarity result between the scene data and scene A can be obtained; for example, the similarity between the scene data and scene A is 50%. If it is desired to identify scene A or scene B, then the neural network can be retrained with samples from scenes A and B, and then the scene data can be input into the trained neural network, obtaining the similarity results between the scene data and scenes A and B; for example, the similarity between the scene data and scene A is 30%, and the similarity with scene B is 60%. This method only obtains the similarity results between the scene data and each scene, resulting in poor accuracy and flexibility. Summary of the Invention

[0003] This application provides a scene recognition method and system to achieve accurate recognition of various scenes.

[0004] To achieve the above objectives, embodiments of this application provide a scene recognition method, comprising: extracting features from scene data to be recognized; and passing the extracted features in parallel through different scene networks to obtain scene recognition results corresponding to each scene network.

[0005] To achieve the above objectives, this application provides a scene recognition system, including: a backbone network configured to extract features from scene data to be recognized; and a multi-head network device including different scene networks, wherein the extracted features are passed in parallel through the different scene networks to obtain scene recognition results corresponding to each scene network.

[0006] The proposed scene recognition method and system in this application pass the extracted features in parallel through different scene networks to obtain the scene recognition results corresponding to each scene network. For example, each scene network outputs whether the scene data is the corresponding scene. Compared with the prior art, which can only obtain the similarity between scene data and each scene, the scheme of this application has higher recognition accuracy. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the structure of the scene recognition system provided in the embodiments of this application;

[0008] Figure 2 This is a schematic diagram of the structure of the scene recognition system provided in the embodiments of this application;

[0009] Figure 3 This is a schematic diagram of the structure of the scene recognition system provided in the embodiments of this application;

[0010] Figure 4 This is a flowchart illustrating the scene recognition method provided in an embodiment of this application;

[0011] Figure 5 This is a flowchart illustrating the scene recognition method provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the shown or described steps may be performed in a different order than the module division in the device or the order shown in the flowchart.

[0013] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0014] like Figure 1 As shown, Figure 1 This is a schematic diagram of the scene recognition system provided in this application embodiment. The system includes, but is not limited to, a backbone network 110 and a multi-head network device 120.

[0015] Backbone network 110 is configured to extract features from the scene data to be identified.

[0016] The backbone network is responsible for feature extraction from the scene data. The scene data includes at least one of scene video data, scene image data, and scene text data. If the scene data is scene text data, the backbone network is a deep neural network pre-trained on the text, and the scene data is processed through the backbone network to obtain vectors representing text features. If the scene data is scene video data or scene image data, the backbone network is a deep neural network pre-trained on ImageNet, and the scene data is processed through the backbone network to obtain vectors representing image features. Optionally, the backbone network is a multi-layer deep neural network with the last few fully connected layers removed from the front part of the network.

[0017] In this embodiment, scene data can optionally be collected using a camera or microphone and stored in memory.

[0018] The multi-head network device 120 includes different scene networks. The extracted features are passed through the different scene networks in parallel to obtain the scene recognition results corresponding to each scene network.

[0019] Each scene network can be configured as a single-layer fully connected or multi-layer perceptron (MLP), and each scene network is called a head. Multiple heads can exist in parallel without affecting each other, and new heads can also be added. Each head outputs a binary classification, that is, whether the scene data belongs to the scene network corresponding to this scene.

[0020] Using existing technologies, for scene data N, the scene recognition result output by the neural network is the approximation degree to each scene, rather than the exact result of whether it is a specific scene. For example, the approximation degree to scene A is 40%, to scene B is 30%, and to scene C is 30%, resulting in poor recognition accuracy. The solution in this embodiment passes the extracted features in parallel through different scene networks, obtaining the scene recognition result corresponding to each scene network. For example, for scene data N, scene network A outputs a recognition result of 1, indicating approximation; scene network B outputs a recognition result of 0, indicating no approximation; and scene network C outputs a recognition result of 0, indicating no approximation. This clearly identifies scene data N as the scene data corresponding to scene network A, resulting in higher recognition accuracy.

[0021] like Figure 2 As shown, Figure 2 This is a schematic diagram of the scene recognition system provided in an embodiment of this application. The system includes, but is not limited to, a positive sample device 210, a backbone network 220, and a multi-head network device 230.

[0022] The positive sample device 210 is configured to output the scene data to be identified to the backbone network.

[0023] The positive sample device collects data from the current scene, obtaining text data, image data, or video data to be identified as scene data.

[0024] Backbone network 220 is configured to extract features from the scene data to be identified.

[0025] The multi-head network device 230 extracts features and passes them in parallel through different scene networks in the multi-head network device to obtain the scene recognition results corresponding to each scene network.

[0026] The scheme in this embodiment passes the extracted features in parallel through different scene networks to obtain the scene recognition results corresponding to each scene network, resulting in higher accuracy of the recognition results.

[0027] like Figure 3 As shown, Figure 3 This is a schematic diagram of the scene recognition system provided in the embodiments of this application. The system includes, but is not limited to, a positive sample device 310, a negative sample generator 320, a scene identification device 330, a backbone network 340, and a multi-head network device 350.

[0028] The positive sample device 310 is configured to output training positive samples to the backbone network.

[0029] The negative sample generator 320 is configured to output training negative samples to the backbone network.

[0030] The training positive samples are selected scene files, and the training negative samples are other scene files besides the selected scene. The difference between scene files and scene data is that scene data refers to the collected scene data directly stored in storage space (e.g., memory), while a scene file is an ordered collection of scene data. For example, reading data from sectors 0 to 127 in memory, or reading the first 128 bytes of the tellme.txt file in the X directory in memory.

[0031] Scene identification device 330 is configured to acquire scene identifiers and output the scene identifiers to the backbone network. The scene identifier is configured to identify the selected scene.

[0032] The backbone network 340 is configured to extract training features of training positive and training negative samples based on the scene identifier.

[0033] The multi-head network device 350 includes different scene networks and a new scene network, with scene identifiers corresponding to the new scene network; the training features of training positive samples and training negative samples are passed through the new scene network to obtain the training recognition result corresponding to the new scene network; based on the training recognition result corresponding to the new scene network, the labels of the training positive samples and the labels of the training negative samples, the weights of the new scene network are determined to obtain the trained scene network.

[0034] Alternatively, the multi-head network device 350 includes different scene networks, with scene identifiers corresponding to existing scene networks in the multi-head network device; the training features are passed through the existing scene networks to obtain the training recognition results corresponding to the existing scene networks; the weights of the existing scene networks are updated based on the training recognition results corresponding to the existing scene networks, the labels of the training positive samples, and the labels of the training negative samples to obtain the updated scene networks.

[0035] Optionally, the multi-head network device can be instructed to recognize scene data, train a new scene network, or update an existing scene network by means of button triggering, key triggering, or sending commands.

[0036] In existing technologies, when a new scene recognition function needs to be added, the neural network is retrained based on the samples corresponding to the original scene recognition function and the samples corresponding to the new scene recognition function. For example, if the original neural network can recognize scene A but cannot recognize scene B, and it is necessary to add the ability to recognize scene B, then the neural network is retrained based on the samples of scene A and scene B. This allows the neural network to recognize the similarity between the scene data and scene A and scene B. For example, the similarity between the scene data and scene A is 30%, and the similarity with scene B is 60%. Using the solution of this embodiment, when a multi-head network device needs to add a new scene recognition function, it is not necessary to retrain the entire multi-head network device; only the network for the new scene needs to be trained. This makes training convenient and quick, and the recognition is flexible and accurate.

[0037] In existing technologies, when scene recognition functionality needs to be updated, the neural network is retrained using samples corresponding to the scene recognition function requiring the update, as well as samples corresponding to other scene recognition functions that do not need updating. For example, if the original neural network can recognize scene A and scene B, and the ability to recognize scene B needs to be updated, then the neural network is retrained based on samples from scene A and the updated scene B. Using the solution in this embodiment, when the multi-head network device needs to update its scene recognition functionality, it is not necessary to retrain the entire multi-head network device; only the network for the scene requiring the update needs to be retrained, making the update convenient and quick.

[0038] like Figure 4 As shown, Figure 4 This is a flowchart illustrating the scene recognition method provided in this application embodiment. The method includes, but is not limited to, steps S410 and S420.

[0039] Step S410: Extract features from the scene data to be identified.

[0040] Scene data includes at least one of the following: scene video data, scene image data, and scene text data. Optionally, the size of the scene data to be identified can be 64*64*3. Compared to scene data of size 32*32*3, scene data of size 64*64*3 has higher resolution and is clearer after dimensionality reduction.

[0041] Step S420: Pass the extracted features in parallel through different scene networks to obtain the scene recognition results corresponding to each scene network.

[0042] Using existing technologies, for scene data N, the scene recognition result output by the neural network is the approximation degree to each scene, rather than the exact result of whether it is a specific scene. For example, the approximation degree to scene A is 40%, to scene B is 30%, and to scene C is 30%, resulting in poor recognition accuracy. The solution in this embodiment passes the extracted features in parallel through different scene networks, obtaining the scene recognition result corresponding to each scene network. For example, for scene data N, scene network A outputs a recognition result of 1, indicating approximation; scene network B outputs a recognition result of 0, indicating no approximation; and scene network C outputs a recognition result of 0, indicating no approximation. This clearly identifies scene data N as the scene data corresponding to scene network A, resulting in higher recognition accuracy.

[0043] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the scene recognition method provided in this application embodiment. The method includes, but is not limited to, steps 510, 520, 530, 540, and 550.

[0044] Step 510: Extract training features from training positive and training negative samples based on scene identifiers.

[0045] The training positive samples are selected scene files, and the training negative samples are other scene files besides the selected scene.

[0046] Step 520: Pass the training features through the network to be trained corresponding to the scene identifier to obtain the training recognition result corresponding to the network to be trained.

[0047] The network to be trained can be an existing scene network or a new scene network.

[0048] Step 530: Based on the training recognition results, the labels of the training positive samples, and the labels of the training negative samples, determine the weights of the network to be trained to obtain the trained scene network.

[0049] The training mechanism of the network to be trained is as follows, wherein:

[0050] Y pr For the obtained output, Y gt For the correct output, W is the weight, X is the input, σ is the activation function (sigmoid), and η is a constant.

[0051] Y pr =σ(WX), where WX≡Z

[0052] The weight update amount is (W = W + ΔW):

[0053]

[0054] Step S540: Extract features from the scene data to be identified.

[0055] Step S550: Pass the extracted features in parallel through different scene networks to obtain the scene recognition results corresponding to each scene network.

[0056] In existing technologies, when a new scene recognition function needs to be added, the neural network is retrained based on samples corresponding to the original scene recognition function and samples corresponding to the new scene recognition function. For example, if the original neural network can recognize scene A but cannot recognize scene B, and it is necessary to add recognition for scene B, then the neural network is retrained based on samples from scene A and scene B. This allows the neural network to recognize the similarity between the scene data and scene A and scene B. For example, the similarity between the scene data and scene A is 30%, and the similarity with scene B is 60%. Using the solution in this embodiment, when a new scene recognition function needs to be added, only the new scene network needs to be trained, making training convenient and quick, and providing flexible and accurate recognition.

[0057] In existing technologies, when scene recognition functionality needs to be updated, the neural network is retrained using samples corresponding to the scene recognition function requiring the update, as well as samples corresponding to other scene recognition functions that do not need updating. For example, if the original neural network can recognize scene A and scene B, and the ability to recognize scene B needs to be updated, then the neural network is retrained based on samples from scene A and the updated scene B. Using the solution in this embodiment, when scene recognition functionality needs to be updated, only the network for the scene requiring the update (the existing scene network) needs to be retrained, making the update convenient and quick.

[0058] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0059] The above description is merely an exemplary embodiment of this application and is not intended to limit the scope of protection of this application.

[0060] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although this application is not limited thereto.

[0061] Embodiments of this application can be implemented by executing computer program instructions through the data processor of a mobile device, for example, in a processor entity, or through hardware, or through a combination of software and hardware. The computer program instructions can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages.

[0062] Any block diagram of logical flow in the accompanying drawings of this application may represent program steps, or may represent interconnected logic circuits, modules, and functions, or may represent a combination of program steps and logic circuits, modules, and functions. The computer program may be stored on memory. Memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as, but not limited to, read-only memory (ROM), random access memory (RAM), optical storage devices and systems (Digital Multifunction Discs, DVDs, or CDs), etc. Computer-readable media may include non-transitory storage media. The data processor may be of any type suitable to the local technical environment, such as, but not limited to, general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (FPGAs), and processors based on multi-core processor architectures.

[0063] A detailed description of exemplary embodiments of this application has been provided above through exemplary and non-limiting examples. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, without departing from the scope of the invention. Therefore, the proper scope of the invention will be determined by the claims.

Claims

1. A scene recognition method, characterized in that: include: Features of the scene data to be identified are extracted through a backbone network; wherein, the scene data includes: scene video data, scene image data and scene text data, and the scene data is collected through a camera and / or microphone; The extracted features are passed in parallel through multiple scene networks using a multi-head network device to obtain scene recognition results for each scene network. Each scene recognition result includes 0 or 1, where 0 indicates that the scene data is not the scene data corresponding to the scene network, and 1 indicates that the scene data is the scene data corresponding to the scene network. Each scene network is configured as a single-layer fully connected or multi-layer perceptron, and multiple scene networks exist in parallel without affecting each other. In the case where the multiple different scene networks include existing scene networks and new scene networks, the new scene network is trained in the following way: training features of training positive samples and training negative samples are extracted based on scene identifiers, where the scene identifiers correspond to the new scene network and are used to identify the selected scene; the training features are passed through the new scene network to obtain the training recognition result corresponding to the new scene network; The new scene network is implemented by adding new heads to the multi-head network device, and the multiple different scene networks correspond to the multiple heads of the multi-head network device; and when it is necessary to add a new scene recognition function, new heads are added to the multi-head network device so that the number of heads of the multi-head network device increases dynamically, and the new scene is recognized by the newly added heads, and the new heads correspond to the new scene network.

2. The method according to claim 1, characterized in that, Before extracting features from the scene data to be identified, the process also includes: Training features of positive and negative training samples are extracted based on scene identifiers; The training features are passed through the network to be trained corresponding to the scene identifier to obtain the training recognition result corresponding to the network to be trained; Based on the training recognition results, the labels of the training positive samples, and the labels of the training negative samples, the weights of the network to be trained are determined, and the trained scene network is obtained.

3. The method according to claim 2, characterized in that, The training positive samples are scene files, and the training negative samples are non-scene files.

4. The method according to any one of claims 1 to 3, characterized in that, The scene data includes at least one of the following: scene video data, scene image data, and scene text data.

5. A scene recognition system, characterized in that, include: The backbone network is configured to extract features from the scene data to be identified; wherein, the scene data includes: scene video data, scene image data and scene text data, and the scene data is collected through a camera and / or microphone; A multi-head network device includes multiple different scene networks. The extracted features are passed in parallel through the different scene networks to obtain the scene recognition results corresponding to each scene network. The scene recognition result corresponding to each scene network includes 0 or 1, where 0 indicates that the scene data is not the scene data corresponding to the scene network, and 1 indicates that the scene data is the scene data corresponding to the scene network. Each scene network is set as a single-layer fully connected or multi-layer perceptron, and multiple scene networks exist in parallel without affecting each other. In the case where the multiple different scene networks include existing scene networks and new scene networks, the new scene network is trained in the following way: training features of training positive samples and training negative samples are extracted based on scene identifiers, where the scene identifiers correspond to the new scene network and are used to identify the selected scene; the training features are passed through the new scene network to obtain the training recognition result corresponding to the new scene network; The new scene network is implemented by adding new heads to the multi-head network device, and the multiple different scene networks correspond to the multiple heads of the multi-head network device; and when it is necessary to add a new scene recognition function, new heads are added to the multi-head network device so that the number of heads of the multi-head network device increases dynamically, and the new scene is recognized by the newly added heads, and the new heads correspond to the new scene network.

6. The system according to claim 5, characterized in that, Also includes: The positive sample device is configured to output the scene data to be identified to the backbone network.

7. The system according to claim 6, characterized in that, Also includes: The scene identification device is configured to acquire scene identifications and output the scene identifications to the backbone network; the negative sample generator is configured to output training negative samples to the backbone network; the positive sample device is further configured to output training positive samples to the backbone network; the multi-head network device also includes a new scene network. The backbone network extracts training features of training positive and training negative samples based on the scene identifier, and the scene identifier corresponds to the new scene network; the training features are passed through the new scene network to obtain the training recognition result corresponding to the new scene network; the weights of the new scene network are determined based on the training recognition result corresponding to the new scene network, the labels of the training positive samples and the labels of the training negative samples, to obtain the trained scene network.

8. The system according to claim 6, characterized in that, Also includes: A scene identification device is configured to acquire scene identifications and output the scene identifications to the backbone network; a negative sample generator is configured to output training negative samples to the backbone network. The positive sample device is further configured to output training positive samples to the backbone network; The backbone network extracts training features of training positive and training negative samples based on the scene identifier, and the scene identifier corresponds to the existing scene network in the multi-head network device. The training features are passed through the existing scene network to obtain the training recognition result corresponding to the existing scene network; the weights of the existing scene network are updated according to the training recognition result corresponding to the existing scene network, the labels of the training positive samples and the labels of the training negative samples to obtain the updated scene network.

9. The system according to any one of claims 5 to 8, characterized in that, The backbone network is configured as a deep neural network.

Images