A model training method, a scene recognition method, and related equipment

Abstract

A model training method, scene recognition method, and related equipment are disclosed to improve the accuracy of scene recognition. The method in this application includes: acquiring a first image; using an object detection model to identify images of target objects in the first image that are unrelated to scene recognition; performing masking processing on the region where the target object is located in the first image to obtain a third image; then using an image generation model to generate multiple sample object images unrelated to scene recognition; combining the sample object images and the third image to obtain a target image; inputting the target image into a first convolutional neural network for training, and inputting the third image into a second convolutional neural network for training, to obtain a scene recognition model. The scene recognition model can reduce the focus on image features that produce differences in the target image and more easily learn the differences between different scene categories, thereby improving the accuracy of scene recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, scene recognition method, and related equipment in the field of computer vision within the application of artificial intelligence. Background Technology

[0002] Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0003] Artificial intelligence applications include computer vision, and scene recognition is an important branch of computer vision technology. Scene recognition refers to the identification (or "classification") of the environment or the environment in which a subject (person or object) is located in an image. It aims to extract and analyze features from scene images to obtain information about the scene, thereby identifying the scene to which the image belongs.

[0004] Most current scene recognition devices use general image recognition models (used for both object recognition and scene recognition) to recognize scene images. However, the accuracy of general image recognition models for scene recognition is limited, and the application scenarios for scene recognition are also limited. Summary of the Invention

[0005] This application provides a model training method, a scene recognition method, and related equipment to improve the accuracy of scene recognition.

[0006] Firstly, this application provides a model training method applied to a training device. The method includes: the training device acquiring a first training dataset, which includes multiple first images, each a scene image, such as an image of an "office" scene, which may include images of objects unrelated to scene recognition; the training device using an object detection model to identify a first region in the first image, the first region being an image region unrelated to scene recognition; then, the training device performing masking on the first region to obtain a third image; the masking process occludes the first region; subsequently, the training device acquiring multiple sample object images generated by an image generation model, the sample object images being images of objects unrelated to scene recognition; the training device replacing the masked first region in the third image with the multiple sample object images to obtain multiple target images; combining these multiple target images expands the number of images in the first training dataset in terms of data volume. Furthermore, regarding the differences between images, for images of the same category, the third image retains background images relevant to scene recognition, while the sample object images generated by the image generation model are used as difference images between the newly synthesized target images. Finally, the training device trains a first convolutional neural network using the target image dataset and a second convolutional neural network using the third image dataset to obtain a scene recognition model, which includes both the first and second convolutional neural networks. In this embodiment, the training device trains the first convolutional neural network using a large number of newly synthesized target images. Introducing images of objects unrelated to scene recognition into scene images of the same category reduces the scene recognition model's focus on features of differing images within the scene images, thereby mitigating the adverse effects of intra-class differences on the classification performance of the scene recognition model. Furthermore, after occluding image regions unrelated to scene recognition, the training device trains the second convolutional neural network using images related to scene recognition (i.e., the third image). The second convolutional neural network more easily learns the differences between different scene categories, thus mitigating the adverse effects of inter-class similarity on the classification performance of the scene recognition model. The scene recognition model obtained by the training device can reduce the negative impact of intra-class differences in scene images of the same category and the negative impact of inter-class similarity between different scene categories on the classification performance of the scene recognition model, thereby improving the accuracy of scene recognition.

[0007] In an optional implementation, the method further includes: a training device inputting a first image into an image recognition model, which is a general image recognition model (used for both image recognition and scene recognition); the training device using the image recognition model to obtain a first classification result and a heatmap of the first image, wherein the heatmap is used to show the region where the target object is located, the image features of the target object are image features unrelated to scene recognition, and the category indicated by the first classification result is a non-scene category or an incorrect scene category; the training device performing masking processing on a second region in the first image other than the first region where the target object is located, i.e., occluding the second region, to obtain a second image (i.e., an image containing only the target object); then, the training device using a second training dataset to train the first model to obtain an object detection model, the second training dataset including multiple sample data, the sample data including input data and output data, wherein the input data is the second image, and the output data is position coordinates, the position coordinates being used to indicate the region where the target object is located. In this embodiment, the training device can determine the region in the first image that has the greatest impact on the image recognition model's classification decision through the heat map of the first image. The heat map can also determine the location of target objects that are not related to scene recognition. The first model (such as a neural network) is trained through the second image to obtain an object detection model. The object detection model is used to identify which part of a scene image is not related to scene recognition, and thus it can also determine which part of a scene image is related to scene recognition.

[0008] In an optional implementation, the method further includes: training the generative adversarial network (GAN) using the second image to obtain an image generation model. The image generation model generates a large number of sample object images unrelated to scene recognition, thereby obtaining target images for training the scene recognition model. These target images, obtained from the multiple sample object images, both expand the number of images in the first training dataset and, for images of the same category, serve as difference images between newly synthesized target images, thus mitigating the adverse effects of intra-class differences on the classification performance of the scene recognition model and improving its performance.

[0009] In one optional implementation, both the target image and the third image correspond to labels of the first category. The step of training a first convolutional neural network using the target image and a second convolutional neural network using the third image can specifically include: the training device extracts image features from the target image through the first convolutional layer of the first convolutional neural network, and extracts image features from the third image through the second convolutional layer of the second convolutional neural network. The image features of the third image are then output to the first convolutional layer for fusion with the image features of the target image. The fused image features are then output to the output layer of the first convolutional neural network, and the first category label is output through the output layer (such as a fully connected layer and a classifier). The image features of the third image extracted by the second convolutional neural network are scene-recognition-related image features in the first image. The second convolutional neural network is equivalent to an attention model, and it fuses the extracted image features into the last convolutional layer of the first convolutional neural network, making the scene recognition model focus more on scene-recognition-related image features. Furthermore, by occluding images of objects (target objects) that are irrelevant to scene recognition, the second convolutional neural network is trained using images relevant to scene recognition. This makes it easier for the second convolutional neural network to learn the differences between different scene categories, thereby reducing the adverse effects of inter-class similarity on the classification performance of the scene recognition model.

[0010] Secondly, embodiments of this application provide a scene recognition method applied to an execution device. The method includes: the execution device acquiring a first scene image to be recognized; then, the execution device using an object detection model to detect a first region in the first scene image containing objects unrelated to scene recognition; the execution device performing masking processing on the first region to obtain a second scene image; subsequently, the execution device inputting the first scene image into a first convolutional neural network in the scene recognition model, inputting the second scene image into a second convolutional neural network in the scene recognition model, and outputting a classification result using the scene recognition model. The first convolutional neural network is trained using a dataset of target images, the second convolutional neural network is trained using a dataset of third images, and the target image is obtained by replacing the first region in the third image with multiple sample object images generated by an image generation model. The third image is obtained by using an object detection model to identify the first region in the first image unrelated to scene recognition, and then performing masking processing on the first region. The first image is an image from the training dataset. In this embodiment, the first convolutional neural network is obtained by learning from the target image, and the target image is obtained by synthesizing the same background image with different differing object images (images of objects unrelated to scene recognition). This reduces the scene recognition model's focus on image features irrelevant to scene recognition in the first scene image, thereby mitigating the negative impact of intra-class differences between scene images of the same category on the model's classification performance. The second convolutional neural network, learned from images relevant to scene recognition, extracts image features relevant to scene recognition from the first scene image, paying more attention to these features. This reduces the negative impact of inter-class similarity between different categories of scene images on the model's classification performance, significantly improving the accuracy of the classification results for the first scene image.

[0011] In one optional implementation, the execution device inputs a first scene image into a first convolutional neural network in the scene recognition model, and inputs a second scene image into a second convolutional neural network in the scene recognition model. The output of the classification result using the scene recognition model can specifically include: the execution device extracts image features of the first scene image through the first convolutional layer of the first convolutional neural network, extracts image features of the second scene image through the second convolutional layer of the second convolutional neural network, and outputs the image features of the second scene image to the first convolutional layer for fusion with the image features of the first scene image, thereby enabling the scene recognition model to focus on global information; the first convolutional neural network outputs the fused image features to the output layer, and the classification result is output through the output layer (fully connected layer and classifier) ​​of the first convolutional neural network.

[0012] In one optional implementation, if the classification result indicates a first scene, and the first scene corresponds to a first noise reduction mode of the headphones; when the executing device is a terminal device, the executing device is connected to the headphones, and the method further includes: the executing device adjusting the noise reduction mode of the headphones to the first noise reduction mode according to the classification result; the executing device can recognize the scene image and automatically adjust the noise reduction mode of the headphones according to the classification result obtained from the scene recognition, without requiring the user to manually set the noise reduction mode of the headphones. Alternatively, when the executing device is a server, the user device is connected to the headphones, and the method further includes: the executing device sending the classification result to the user device, the classification result being used to trigger the user device to adjust the noise reduction mode of the headphones to the first noise reduction mode. In this embodiment, the executing device can recognize the scene image and send the classification result to the user device, thereby enabling the user device to automatically adjust the noise reduction mode of the headphones according to the classification result obtained from the scene recognition, without requiring the user to manually set the noise reduction mode of the headphones.

[0013] In an optional implementation, if the classification result indicates a first scene, and the first scene corresponds to a first volume value; when the executing device is a terminal device, the method further includes: the executing device adjusting its system volume to the first volume value according to the classification result. In this embodiment, the executing device can adaptively adjust the system volume value according to the classification result of the scene image, eliminating the need for the user to frequently adjust the phone's system volume value based on different environments. Alternatively, when the executing device is a server, the method further includes: the executing device sending the classification result to the user device, the classification result triggering the user device to adjust its system volume to the first volume value, thereby enabling the user device to automatically adjust its phone's system volume value according to the classification result obtained from scene recognition, eliminating the need for the user to manually adjust the phone's system volume value and improving the user experience.

[0014] In one optional implementation, acquiring the first scene image to be identified may include: the execution device receiving the first scene image to be identified sent by the user device; or, the execution device acquiring the first scene image to be identified through a camera or image sensor.

[0015] Thirdly, embodiments of this application provide a model training apparatus, including:

[0016] The acquisition module is used to acquire the first training dataset, which includes multiple first images;

[0017] The processing module is used to identify a first region in a first image using an object detection model. The first region is an image region unrelated to scene recognition. The first region is masked to obtain a third image. Multiple sample object images generated by an image generation model are obtained. The sample object images are images of objects unrelated to scene recognition. The multiple sample object images are used to replace the masked first region in the third image to obtain multiple target images. A first convolutional neural network is trained using the target image dataset, and a second convolutional neural network is trained using the third image dataset to obtain a scene recognition model, which includes the first convolutional neural network and the second convolutional neural network.

[0018] In an optional implementation, the processing module is further configured to input the first image into an image recognition model, and use the image recognition model to obtain a first classification result and a heatmap of the first image. The heatmap is used to show the region where the target object is located. The image features of the target object are image features that are unrelated to scene recognition. The first classification result indicates a non-scene category or an incorrect scene category. The module then performs masking processing on a second region in the first image other than the region where the target object is located, to obtain a second image. Finally, the module trains the first model using a second training dataset to obtain an object detection model. The second training dataset includes multiple sample data, which includes input data and output data. The input data is the second image, and the output data is position coordinates, which are used to indicate the region where the target object is located.

[0019] In an alternative implementation, the processing module is also used to train the generative adversarial network (GAN) using the second image to obtain an image generation model.

[0020] In one optional implementation, both the target image and the third image correspond to the label of the first category; the processing module is further configured to extract image features of the target image through the first convolutional layer of the first convolutional neural network, extract image features of the third image through the second convolutional layer of the second convolutional neural network, and output the image features of the third image to the first convolutional layer for fusion with the image features of the target image; and output the label of the first category through the output layer of the first convolutional neural network based on the fused image features.

[0021] Fourthly, embodiments of this application provide a scene recognition device, including:

[0022] The acquisition module is used to acquire the first scene image to be identified;

[0023] The processing module is used to detect a first region in a first scene image containing objects unrelated to scene recognition using an object detection model; perform masking processing on the first region to obtain a second scene image; input the first scene image into a first convolutional neural network in a scene recognition model, input the second scene image into a second convolutional neural network in a scene recognition model, and output classification results using the scene recognition model. The first convolutional neural network is trained using a dataset of target images, and the second convolutional neural network is trained using a dataset of third images. The target images are obtained by replacing the first region in the third image with multiple sample object images generated by an image generation model. The third image is obtained by masking the first region in the first image after identifying it using the object detection model. The first image is an image from the training dataset.

[0024] In an optional implementation, the processing module is further configured to extract image features of the first scene image through the first convolutional layer of the first convolutional neural network, extract image features of the second scene image through the second convolutional layer of the second convolutional neural network, and output the image features of the second scene image to the first convolutional layer for fusion with the image features of the first scene image; and output a classification result based on the fused image features through the output layer of the first convolutional neural network.

[0025] In one optional implementation, the device further includes a sending module; if the classification result indicates a first scenario, the first scenario corresponds to a first noise reduction mode of the headphones; the processing module is further configured to adjust the noise reduction mode of the headphones to the first noise reduction mode according to the classification result; or, the sending module is configured to send the classification result to the user device, the classification result being used to trigger the user device to adjust the noise reduction mode of the headphones to the first noise reduction mode.

[0026] In one optional implementation, if the classification result indicates a first scenario, and the first scenario corresponds to a first volume value, the processing module is further configured to adjust the system volume of the execution device to the first volume value according to the classification result; or, the sending module is further configured to send the classification result to the user device, and the classification result is used to trigger the user device to adjust the system volume of the user device to the first volume value.

[0027] In one alternative implementation, the acquisition module is further specifically used to: receive a first scene image to be identified sent by a user device; or, acquire the first scene image to be identified via a camera or image sensor.

[0028] Fifthly, embodiments of this application provide an electronic device, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the electronic device performs the method as described in any one of the first aspects above; or, when the program or instructions are executed by the processor, the electronic device performs the method as described in any one of the second aspects above.

[0029] Sixthly, embodiments of this application provide a computer program product, which includes computer program code. When the computer program code is executed by a computer, it causes the computer to implement the method described in any one of the first aspects above; or, when the computer program code is executed by a computer, it causes the computer to implement the method described in any one of the second aspects above.

[0030] In a seventh aspect, embodiments of this application provide a computer-readable storage medium for storing a computer program or instructions, which, when executed, cause a computer to perform the method as described in any one of the first aspects above; or, when executed, the computer program or instructions cause a computer to perform the method as described in any one of the second aspects above. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the main artificial intelligence architecture in the embodiments of this application;

[0032] Figure 2A and Figure 2B This is a schematic diagram of the system architecture in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the original image and its heatmap;

[0034] Figure 4 This is a schematic diagram illustrating the steps for training the object detection model and the image generation model in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of obtaining a second image after processing the first image mask in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the architecture of the scene recognition model in the embodiments of this application;

[0037] Figure 7 This is a flowchart illustrating the steps for training the scene recognition model in this embodiment of the application.

[0038] Figure 8 This is a schematic diagram of obtaining a third image after masking the first image in an embodiment of this application;

[0039] Figure 9 This is an architecture diagram of the object detection model and scene recognition model in the embodiments of this application;

[0040] Figure 10 This is a flowchart illustrating the steps of one embodiment of a scene recognition method according to this application.

[0041] Figure 11A , Figure 11B and Figure 11C This is a schematic diagram of the setting interface for the correspondence between headphone noise reduction modes and scenes in an embodiment of this application;

[0042] Figure 12 This is a schematic diagram illustrating the modified correspondence between the scene and the noise reduction mode in the embodiments of this application;

[0043] Figure 13 This is a schematic diagram of the setting interface for the correspondence between scene and system volume values ​​in an embodiment of this application;

[0044] Figure 14 This is a schematic diagram of one embodiment of a model training device according to the present application.

[0045] Figure 15 This is a schematic diagram of the structure of one embodiment of the neural network processor in this application.

[0046] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0047] Figure 17 This is a schematic diagram of the structure of one embodiment of a scene recognition device according to the present application.

[0048] Figure 18 This is a schematic diagram of the structure of another electronic device in an embodiment of this application. Detailed Implementation

[0049] This application relates to the field of computer vision, an application area of ​​artificial intelligence, and particularly to scene recognition within computer vision. First, the main framework of artificial intelligence will be explained.

[0050] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0051] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0052] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0053] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0054] (1) Infrastructure:

[0055] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0056] (2) Data

[0057] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0058] (3) Data processing

[0059] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0060] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0061] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0062] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0063] (4) General ability

[0064] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0065] (5) Smart Products and Industry Applications

[0066] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0067] Scene recognition is an important branch of computer vision. It refers to the identification (or "classification") of the environment or the surroundings of a subject (person or object) within an image. Compared to image recognition centered on the subject (or "object"), scene recognition focuses on the global information of the image. This makes it easy for recognition devices to mistakenly identify objects unrelated to the environment as key features for scene recognition, leading to two main challenges. First, scene images of the same category exhibit differences (i.e., intra-class differences), which may arise from differences between objects unrelated to the environment. For example, image A shows Xiaoming wearing a mask at the airport, while image B shows him without a mask at the airport. Both images depict an "airport" scene, but recognition devices are more likely to mistakenly identify a face in image A (unrelated to the environment) as a key feature, resulting in an incorrect classification (e.g., "hospital"). Second, scene images of different categories exhibit similarities (i.e., inter-class similarities), which may also arise from the similarities between objects unrelated to the environment. For example, image C shows the interior of a high-speed train including seats, while image D shows the interior of an airport including seats. Recognition devices are more likely to use seats as a key feature for scene recognition, performing scene recognition on image D and using the seats in image D as the key feature, resulting in an incorrect classification result (e.g., "high-speed train"). Intra-class differences and inter-class similarities lead to a decrease in the accuracy of scene recognition.

[0068] To address the aforementioned problems, this application provides a scene image recognition method. This method reduces intra-class differences and inter-class similarities in scene images, thereby improving the accuracy of scene recognition. Please refer to... Figure 2AAs shown, this application embodiment provides a system architecture. A data acquisition device 210 is used to acquire images, storing the acquired images as training data in a database 230. A training device 220 generates an object detection model and a scene recognition model based on the image data maintained in the database 230. The object detection model is used to detect regions in the image to be recognized that are "unrelated to scene (environment) recognition." The scene recognition model is used to recognize the scene image to be recognized. The training device 220 is implemented by one or more servers; optionally, it is implemented by one or more terminal devices. An execution device 240 obtains the object detection model and scene recognition model from the training device 220 and loads them into the execution device 240. After acquiring the scene image to be recognized, the execution device 240 can use the object detection model and scene recognition model to recognize the scene image and obtain a classification result. The execution device 240 is a terminal device; for example, it includes, but is not limited to, mobile phones, personal computers, tablets, wearable devices (e.g., watches, wristbands, VR / AR devices), and in-vehicle terminals. Optionally, please refer to... Figure 2B As shown, the system architecture also includes user device 250, which includes, but is not limited to, mobile phones, personal computers, tablets, wearable devices (such as watches, wristbands, VR / AR devices), and in-vehicle terminals. Execution device 240 is implemented by one or more servers. User device 250 can interact with execution device 240 through any communication mechanism or standard communication network, such as a wide area network, local area network, point-to-point connection, or any combination thereof. User device 250 is used to acquire scene images to be recognized and send these images to execution device 240. Execution device 240 receives the scene images from user device 250, uses an object detection model and a scene recognition model to recognize the scene images, and obtains a classification result. Execution device 240 then sends the classification result back to user device 250. Optionally, training device 220 and execution device 240 can be the same device; for example, a server (or server cluster) can be used to implement both the functions of training device 220 and execution device 240.

[0069] This application provides a model training method applied to the training device in the aforementioned system architecture. The training device acquires a first training dataset, which includes multiple first images. An object detection model is used to identify images of target objects in the first images that are unrelated to scene recognition. The training device performs masking on the regions containing the target objects in the first images to obtain a third image (i.e., containing only images related to scene recognition). The training device uses an image generation model to generate a large number of sample object images unrelated to scene recognition. These sample object images and the third image are combined to obtain a combined target image. The training device inputs the combined target image into a first convolutional neural network for training, and inputs the third image into a second convolutional neural network for training, resulting in a scene recognition model. The scene recognition model includes both the first and second convolutional neural networks. By training the first convolutional neural network with a large number of newly synthesized target images, object images unrelated to scene recognition are introduced into scene images of the same category. This reduces the scene recognition model's focus on image features that cause differences in the target images, thereby reducing the negative impact of intra-class differences in scene images of the same category on the classification performance of the scene recognition model. Furthermore, by training the second convolutional neural network with images related to scene recognition, the second convolutional neural network can more easily learn the differences between different scene categories, thereby reducing the negative impact of inter-class similarity between different scene categories on the classification performance of the scene recognition model and improving the accuracy of scene recognition.

[0070] Furthermore, this application provides a scene recognition method applied to the execution device in the aforementioned system architecture. The execution device acquires a first scene image to be recognized via a camera and / or image sensor. Then, the execution device uses an object detection model obtained from the aforementioned training device to detect a first region in the first scene image containing objects unrelated to scene recognition. The execution device performs masking processing on the first region to obtain a second scene image. The execution device inputs the first scene image and the second scene image into the aforementioned training device to obtain a scene recognition model, and uses the scene recognition model to output a classification result.

[0071] To better understand this application, the terms used in this application will first be explained.

[0072] Scene recognition refers to classifying the environment or setting of an object (person or thing) as depicted in an image. Scene image categories can include, but are not limited to, "airport," "high-speed rail," "hospital," "office," "coffee shop," etc. Optionally, scene image categories can also include, for example, "indoor scene," "outdoor scene," "noisy scene," "quiet scene," "listening scene," etc. The category of the scene image is configured according to the specific application scenario and is not specifically limited.

[0073] Intra-class diversity of scene images refers to the differences between scene images of the same category, causing images with high intra-class diversity to be easily misclassified into other categories. For example, an image of an office scene that includes a "face" image may be misclassified into a category other than "office" due to the introduction of the difference information (the image of a face).

[0074] Inter-class similarity of scene images refers to the phenomenon where scene images of different categories contain similar object images, causing them to be misclassified into the same category. For example, images of the interior of a high-speed train and images of the interior of an airport both contain "chairs." Due to the similarity of "chairs," these images are easily classified into the same category, such as both being classified as "high-speed train" or both as "airport."

[0075] A gradient-weighted class activation map (CAM) is a tool used to visualize convolutional neural networks (CNNs). It describes which local location in an image led the CNN to make its final classification decision. A CAM includes a two-dimensional feature grid associated with the output class, where the position of each network element represents the importance of that class. See also... Figure 3 As shown, Figure 3 This is a schematic diagram of the original image and its heatmap, presenting the similarity between each grid position in the image and the classification result in the form of a heatmap. Figure 3 The image contains a cat and a dog. The CNN classifies the image as "dog". As can be seen from the heatmap, the CNN identified the feature of "dog's face position", that is, it used the features of the dog's face as the key feature for classification and classified the image as "dog".

[0076] The basic principle of heatmaps is briefly explained below. An image is input into a convolutional neural network (CNN), which extracts image features. Global average pooling (GAP) is performed on the last feature map of the CNN model, calculating the mean of each channel. Then, the gradient of the output of the category with the largest value relative to the last feature map is calculated, and this gradient is visualized on the original image. Intuitively, a heatmap shows which part of the high-level features extracted by the CNN has the greatest impact on the final classification decision.

[0077] Generative adversarial networks (GANs) are used to generate sample data. In this application, a GAN is used to generate images of objects in an image that are unrelated to scene recognition. A GAN consists of a generative model (G) and a discriminative model (D). The generative model generates a sample that resembles the real training data as closely as possible. The discriminative model is a binary classifier used to estimate the probability that a sample comes from the real training data. If the discriminative model estimates that the sample comes from the real training data, it outputs a high probability. If the discriminative model estimates that the sample comes from the sample generated by the generative model, it outputs a low probability. It can be understood that the goal of the generative model is to generate samples that are identical to real samples, making it impossible for the discriminative model to distinguish them. The goal of the discriminative model is to detect the samples generated by the generative model. Through the adversarial interaction and game between G and D, the samples generated by the GAN are made closer to real samples, thus obtaining a large amount of sample data.

[0078] This application consists of two parts: the first part is the model training process, and the second part is the execution (inference) process. The model training process will be described first below.

[0079] I. Model Training Process. The training process is primarily executed by the training equipment. The model training process mainly involves three models: an object detection model, an image generation model, and a scene recognition model.

[0080] Please see Figure 4 As shown, the training process of the object detection model and the image generation model will be explained first.

[0081] S10. The training device acquires the first training dataset, which includes multiple first images (or "original images").

[0082] The data acquisition device acquires images and stores them in a database. The training device obtains a first training dataset from the database. For example, the data acquisition device is a device with an image sensor, such as a camera, camcorder, or mobile phone. The first training dataset includes a large number of images of different categories. For example, A1 - "Airport", A2 - "High-speed Rail", A3 - "Subway", A4 - "Office", A5 - "Hospital", etc., and the specific categories are not limited. It should be understood that there are multiple ways to classify the images in the first training dataset according to different needs, and the specific classification varies depending on the specific application scenario. It should be noted that in this application, in order to distinguish between the original images in the first training dataset and the images after processing the original images, the original images are referred to as "first images". The image after masking the "images related to scene recognition" in the first image is called "second images" (only images of objects unrelated to scene recognition are retained). The image after masking the "images of objects unrelated to scene recognition" in the first image is called "third images" (only images related to scene recognition are retained).

[0083] S11. The training device inputs the first image into the image recognition model and uses the image recognition model to obtain the first classification result of the first image and the heat map of the first image.

[0084] The heatmap is used to show the area where the target object is located. The image features of the target object are image features unrelated to scene recognition. The first classification result indicates a non-scene category or an incorrect scene category. The image recognition model is a general object recognition model used to identify target objects (or "target objects") in an image. For example, the first image is a scene image of "a person working in an office." When this first image is input into a general image recognition model, the image recognition model outputs the first classification result as "person." The heatmap of the first image reveals the area (i.e., the area where the face is located) that has the greatest influence on the image recognition model's classification decision. The general image recognition model focuses more on the image features of the main subject in the image; therefore, the category indicated by the output classification result (such as "person") is a non-scene category or an incorrect scene category.

[0085] The purpose of step S11 is to obtain a heat map of the first image. The heat map can determine the location of target objects (such as "faces") that are not related to scene recognition, so that an image containing only the target objects can be obtained (step S12 below), or an image of the remaining area in the first image after the target objects are occluded can be obtained (step S22 below).

[0086] S12. The training device performs masking on the second region in the first image, excluding the first region where the target object is located, to obtain the second image (i.e., an image containing only the target object).

[0087] The first image is any image in the first training dataset. Each image in the first training dataset is processed through steps S11 and S12 described above to obtain the second training dataset, which contains multiple second images. It should be noted that, in this application, to distinguish between "regions containing objects unrelated to scene recognition" and "regions related to scene recognition" in an image, the "regions containing objects unrelated to scene recognition" are referred to as the "first region," and the "regions related to scene recognition" are referred to as the "second region." For an example, please refer to [link to example]. Figure 5 As shown, Figure 5 Taking the first image A as an example, the first image A is an image of an office scene including a "face". The "face" 501 is a target object unrelated to the "office" scene recognition, and the first region 502 where the "face" is located is a region unrelated to scene recognition. The region in the first image A other than the first region 502 is the second region 503. The second region 503 is masked (e.g., the pixel value of the second region is set to 0) to obtain the second image A.

[0088] S13. The training device trains the first model using the second training dataset to obtain an object detection model. The object detection model is used to detect the first region containing objects in the input first image that are unrelated to scene recognition. The first model can be a neural network model.

[0089] The second training dataset includes multiple sample data, each of which includes input data and output data. The input data is the second image, and the output data is the position coordinates, which are used to indicate the rectangular area where the target object is located.

[0090] Based on the above steps S10-S13, the object detection model is obtained.

[0091] S14. The training device trains the GAN network using the second image to obtain an image generation model. The training device then uses the image generation model to generate multiple sample object images of the same category as the target object.

[0092] The process of optimizing (or training) the GAN network using a second image is as follows: With the generative model (G) fixed, the discriminative model (D) is optimized. When the second image (i.e., real data) is input, D optimizes its network structure to output 1. When data generated by G is input, D optimizes its network structure to output 0. With D fixed, G optimizes its network to output samples that are as similar as possible to the real data, and ensures that after the generated samples are discriminated by D, D outputs a high probability value. The training processes of G and D alternate; this adversarial process makes the images generated by G increasingly realistic, and D's ability to "detect fakes" becomes stronger.

[0093] For example, if the second image A is a "face" image, the image generation model will generate a large number of "face" images. These "face" images generated by the image generation model are not actually the faces of real people, but rather images created by the model based on its learning from the second image A, possessing all the features of a real "face." Similarly, if the second image B is a "chair" image, the image generation model will generate a large number of "chair" images, and so on.

[0094] Through steps S10-S12 and S14 described above, an image generation model is obtained. Steps S13 and S14 are not time-dependent; they can be executed synchronously, meaning the image generation model and object detection model are obtained simultaneously. Alternatively, S13 can be executed before S14, meaning the object detection model is obtained first, followed by the image generation model. Or, S13 can be executed after S14, meaning the image generation model is obtained first, followed by the object detection model.

[0095] The training process of the scene recognition model is explained below. First, the architecture of the scene recognition model is described. Please refer to [link / reference needed]. Figure 6 As shown, the scene recognition model includes two branch structures (or a backbone structure and a branch structure), which are two parallel sub-networks. To distinguish between the two sub-networks, they are referred to as a first convolutional neural network and a second convolutional neural network, respectively. The first convolutional neural network includes multiple first convolutional layers, a first fully connected layer, and a classifier. The first convolutional layers, the first fully connected layer, and the classifier are connected sequentially. The second convolutional neural network includes multiple second convolutional layers and second fully connected layers. The second fully connected layer is connected to the last first convolutional layer. In this embodiment, to distinguish between the convolutional layers and fully connected layers in the first and second convolutional neural networks, the convolutional layers in the first convolutional neural network are referred to as "first convolutional layers," the convolutional layers in the second convolutional neural network are referred to as "second convolutional layers," the fully connected layers in the first convolutional neural network are referred to as "first fully connected layers," and the fully connected layers in the second convolutional neural network are referred to as "second fully connected layers."

[0096] Please see Figure 7 As shown, the training process of the scene recognition model is as follows: steps S20-S25.

[0097] S20. The training device acquires the first training data set. The first training data set includes multiple first images (or "raw images").

[0098] Please refer to the above for this step. Figure 5 The explanation of step S10 in the corresponding example will not be repeated here.

[0099] S21. The training device inputs the first image into the object detection model and uses the object detection model to identify the first region in the first image. The first region is an image region unrelated to scene recognition. The object detection model is as described above. Figure 4 The object detection model obtained from steps S11-S13 in the corresponding example.

[0100] For example, the first image C is a foreground image of a human face and a background image of an office scene. The first image C is input into an object detection model, which outputs four coordinate points indicating a first region including the human face. This first region is unrelated to scene recognition.

[0101] S22. The training device performs masking on the first region to obtain the third image.

[0102] For example, please refer to Figure 8 As shown, the region in the first image C that includes the face 501 is the first region 502, and the region in the first image C other than the first region 502 is the second region 503. A masking process is applied to the first region 502 to obtain the third image. The purpose of the masking process is to occlude the first region 502, for example, by setting the pixel value of the first region to "0", so that the third image only contains the image of the second region 503, that is, the third image mainly contains images related to scene recognition.

[0103] S23. The training device acquires multiple sample object images generated by the image generation model. The sample object images are images of objects unrelated to scene recognition.

[0104] The image generation model generates a large number of sample object images based on objects "unrelated to scene recognition" in each of the first images in the first training set. Please refer to the above for this step. Figure 4 The explanation of S14 in the corresponding example will not be repeated here.

[0105] S24. The training device replaces multiple sample object images with the mask-covered area in the third image to obtain multiple target images.

[0106] The third image can be understood as an image that includes only the background relevant to scene recognition, after objects unrelated to scene recognition (also called "interference images") have been occluded. For example, consider an image from a scene image of the first category, where the first category can be any of several scene categories. Taking the "office" category as an example, for a scene image (first image A) in the "office" category, the training device masks the region corresponding to the "face" interference image in first image A to obtain third image A. Then, a large number of different "face" images generated by the image generation model are used to replace the masked region in third image A, and combined to obtain multiple target images (the combined new images). The label corresponding to the combined multiple target images is still "office". For example, for another scene image (first image B) in the "office" category, the region corresponding to the interfering image "chair" in first image B is masked to obtain third image B. Third image B contains the masked region. Then, a large number of "chair" images generated by the image generation model are used to replace the masked region in third image B, combining them to obtain multiple target images. The label for these multiple target images remains "office". Optionally, the training device can also replace the "chair" images generated by the image generation model with the masked region in third image A, combining them to obtain multiple target images. Alternatively, the training device can replace the "face" images generated by the image generation model with the masked region in third image B, combining them to obtain multiple target images.

[0107] In this step, the third image and the sample object images generated by the image generation model are combined to obtain a large number of new target images. Each first image in the first training dataset is processed through steps S21 and S22, and then multiple sample object images generated by the image generation model are combined with the third image respectively. This combination of multiple target images expands the number of images in the first training dataset in terms of data volume. Furthermore, regarding the differences between images, for images of the same category, the third image retains the background image relevant to scene recognition, while the sample object images generated by the image generation model are used as difference images between the newly synthesized scene images. The label corresponding to the new target images obtained by combination is still the first category (e.g., office category), and the target images are used as training data for the scene recognition model. Training a scene recognition model with multiple target images, where the target images have the same (or similar) background, reduces the scene recognition model's attention to (or sensitivity to) intra-class differences in scene images of the same category. This makes the scene recognition model pay less attention to intra-class differences (such as different foreground images) in scene images of the same category, and thus pay more attention to intra-class similarities (such as the same background image) in scene images of the same category, thereby improving the classification accuracy of the scene recognition model.

[0108] S25. The training device inputs the target image into the first convolutional neural network, trains the first convolutional neural network using the dataset of the target image, and inputs the third image into the second convolutional neural network, trains the second convolutional neural network using the dataset of the third image, and obtains a scene recognition model, which includes the first convolutional neural network and the second convolutional neural network.

[0109] When training the scene recognition model, the training data for the first convolutional neural network and the training data for the second convolutional neural network are different. That is, the training data for the first convolutional neural network is a large number of target images (i.e., new scene images obtained by combining them), while the training data for the second convolutional neural network is a third image (i.e., the image after masking out objects unrelated to scene recognition from the original image).

[0110] For example, an original image A of an office scene includes a foreground image (a face) and a background image. The "face" in this original image A is an object unrelated to scene recognition. Occluding the area containing the "face" yields image B (the third image), which serves as input to the second convolutional neural network. Simultaneously, replacing the occluded area in image B with other object images unrelated to scene recognition (such as a face generated by an object generation model, or a chair), results in multiple target images (such as images C, D, and F, etc.). These multiple target images then serve as input to the first convolutional neural network. The similarities between the target images and the third image are: the background images of the target images (images C, D, and F) are identical, all originating from the original image A; the image information of the third image (image B) also originates from the original image A. The differences between the target images and the third image are: the target images (images C, D, and F) include both scene-recognition-related and scene-unrelated object images; the third image (image B) only contains scene-recognition-related images. That is, during the training of the scene recognition model, the two branches of the scene recognition model receive two training data simultaneously.

[0111] The two branch structures are explained below. For the first branch structure, the convolutional layer of the first convolutional neural network (also called the "first convolutional layer") is used to extract image features from the target image. The first convolutional neural network can be divided into multiple stages of convolutional feature extraction operations. For example, these multiple stages of convolutional feature extraction operations, in order from left to right (from shallow to high layers), can be denoted as "block_1", "block_2", ..., "block_n". The image size corresponding to each stage is different, and the size of the image features decreases from "block_1" to "block_n". Taking n as an example (5), the size of block_1 is 224×224×64; the size of block_2 is 112×112×128; the size of block_3 is 56×56×256; the size of block_4 is 28×28×512; and the size of block_5 is 14×14×512. The feature maps of the first two convolutional layers (block_n-2 and block_n-1) of the last convolutional layer (block_n) are pooled (e.g., average pooling) to change the size of these two blocks. The features of block_n-2 and block_n-1 are then fused into the image features of the last block_n, thus achieving multi-scale feature fusion, i.e., fusion of high-level and shallow-level features. This allows the scene recognition model to focus more on global features. Furthermore, by training the first convolutional neural network with a large number of newly synthesized target images, object images unrelated to scene recognition are introduced into scene images of the same category. This reduces the scene recognition model's focus on features of differing images within the scene, thereby mitigating the adverse effects of intra-class differences on the classification performance of the scene recognition model. It should be noted that the "feature fusion" described in this embodiment can be achieved by concatenating, summing, or weighted averaging image features (or feature maps).

[0112] Simultaneously, for the second branch structure, the convolutional layer of the second convolutional neural network (also called the second convolutional layer) is used to extract image features from the third image. These image features pass through a fully connected layer (the second fully connected layer), and the output of this layer is fused to the last convolutional layer (block_n) of the first convolutional neural network. The fused features then pass through the fully connected layer (the first fully connected layer) of the first convolutional neural network and a classifier to output classification results (labels). The image features extracted by the second convolutional neural network from the third image are scene-recognition-related features from the original image. The second convolutional neural network is equivalent to an attention model, fusing the extracted features into the last convolutional layer of the first convolutional neural network, making the scene recognition model focus more on scene-recognition-related image features. Furthermore, by occluding images of objects unrelated to scene recognition, and training the second convolutional neural network with images relevant to scene recognition, it is easier for the second convolutional neural network to learn the differences between different scene categories, thereby mitigating the adverse effects of inter-class similarity on the classification performance of the scene recognition model.

[0113] Optionally, the shallow features extracted by the first and second convolutional neural networks are similar. To reduce the number of model parameters and the model size, the first and second convolutional neural networks can reuse some convolutional layers. For example, both the first and second convolutional neural networks include 20 convolutional layers. The first and second convolutional neural networks can reuse the first 8 convolutional layers, while the 9th to 20th convolutional layers in the first convolutional neural network (such as "convolutional layer 9a-convolutional layer 20a") and the 9th to 20th convolutional layers in the second convolutional neural network (such as "convolutional layer 9b-convolutional layer 20b") are deployed separately.

[0114] II. The Execution Process of Scene Recognition. The execution subject of the scene recognition process is the execution device. For example, the execution device can be a mobile phone.

[0115] Please refer to Figure 9. Figure 9 This is an architecture diagram of the object detection model and the scene recognition model. The terminal device contains both the scene recognition model and the object detection model. The object detection model is used to detect regions in the input image containing objects unrelated to scene recognition, while the scene recognition model is used to classify the scene in the image to be recognized. Please refer to the above for the architecture of the scene recognition model. Figure 6 The corresponding architecture details are not elaborated here. Please refer to [link / reference needed]. Figure 10 As shown, Figure 10 This is a flowchart illustrating the steps of a scene recognition method.

[0116] Step S30: The execution device acquires the first scene image to be identified through the camera.

[0117] The camera can be activated by the user, for example, when a user clicks the camera icon, the device receives the user's click and activates the camera to capture an initial image of the scene. Alternatively, the camera can be activated by an application (app), for example, during a video call in an instant messaging app, the camera activates and captures an initial image of the scene. Or, the camera can automatically activate upon receiving a scene recognition request; for example, when the device detects a change in its position using sensors, the scene it is in may also be in may have changed, requiring re-recognition of the scene, thus activating the camera to capture an initial image of the scene. Taking a mobile phone as an example, the camera can be a front-facing camera or a rear-facing camera; the specific type is not limited.

[0118] Step S31: The execution device uses an object detection model to detect the first region in the first scene image where objects unrelated to scene recognition are located.

[0119] The object detection model used in this step is as described above. Figure 4 The corresponding example uses the object detection model trained in steps S11-S13. The execution device inputs the first scene image to be recognized into the object detection model, which outputs position coordinates to indicate the first region. For example, the position coordinates are 4 pixels, indicating a rectangular region. The object image within this rectangular region (i.e., the first region) is an image unrelated to scene recognition. For example, the first scene image is an image of an office scene, and the central region of the first scene image is an image of a "face." The object detection model detects the "face" as being located in the first region.

[0120] Step S32: The device performs masking on the first area to obtain a second scene image.

[0121] The purpose of masking is to occlude the first region, ensuring that the second scene image contains only images relevant to scene recognition, excluding those unrelated to scene recognition. For example, setting the pixel value of the rectangular region containing the "face" to "0" occludes that region, resulting in the second scene image.

[0122] Step S33: The execution device inputs the first scene image and the second scene image into the scene recognition model, and outputs the classification result using the scene recognition model.

[0123] The scene recognition model includes a first convolutional neural network (CNN) and a second CNN. The first CNN receives a first scene image and extracts first image features from the first scene image. The second CNN receives a second scene image and extracts second image features from the second scene image. The second image features are then output to the last convolutional layer of the first CNN to fuse the second image features with the first image features. The first CNN then outputs the fused image features to an output layer (including a first fully connected layer and a classifier), and the classification result is output through the output layer.

[0124] In this embodiment, the first convolutional neural network is obtained by learning from a target image, which is synthesized from a background image with different, dissimilar object images (object images unrelated to scene recognition). This reduces the scene recognition model's focus on image features unrelated to scene recognition in the first scene image, thereby mitigating the negative impact of intra-class differences between scene images of the same category on the classification performance of the scene recognition model. The second convolutional neural network is obtained by learning from images related to scene recognition, enabling the scene recognition image to extract image features relevant to scene recognition, and focusing more on image features related to scene recognition in the first scene image. This reduces the negative impact of inter-class similarity between scene images of different categories on the classification performance of the scene recognition model, thus significantly improving the accuracy of the classification result of the first scene image to be recognized.

[0125] The scene recognition method provided in this application embodiment can be applied to many specific application scenarios. In the first application scenario, the mobile phone can adjust the noise reduction mode of the headphones according to the classification result of the scene image, as described in S34A below. In the second application scenario, the mobile phone can adjust the volume according to the classification result of the scene image, as described in S34B below. The application scenarios to which the classification result of the first scene image can be applied are described below.

[0126] Explanation of the first application scenario. First, a brief explanation of the headphone's noise cancellation modes. The headphones are pre-configured with two or more noise cancellation modes, such as Mode 1 (or "Deep Noise Cancellation Mode"), Mode 2 (or "Natural Noise Cancellation Mode"), and Mode 3 (or "Transparency Mode" or "Monitoring Mode"). The general principle of headphone noise cancellation is that the headphones pick up ambient noise through a microphone, and then generate anti-noise waves to cancel out external sounds, achieving full or partial noise cancellation before the external sound reaches the user's ears. Mode 1 controls the headphones to activate Deep Noise Cancellation, blocking most of the surrounding noise. The roar of public transportation such as airports, trains, and subways, as well as the noise of noisy urban environments, can easily cause feelings of irritability and agitation. Switching the headphones to Deep Noise Cancellation Mode can effectively isolate ambient noise. Mode 2 controls the headphones to activate Normal Noise Cancellation, blocking a small portion of the surrounding noise. When the headphones activate the second mode, the user can hear some ambient sounds. This mode is suitable for everyday environments such as restaurants, streets, and shopping malls, filtering out some background noise while still allowing the user to perceive surrounding sounds. The third mode reduces ambient noise while still allowing the user to hear human voices and other speech, preventing the user from missing important work information. The three noise cancellation modes described above are merely illustrative and not limiting.

[0127] This section explains the current method for switching noise cancellation modes in headphones. When headphones are connected to a phone, the user needs to set the noise cancellation mode through the phone's settings interface, such as "Settings" - "General" - "Noise Cancellation Mode" - "Deep Noise Cancellation". For example, if a user is on the subway and wants to adjust the noise cancellation mode, they need to open the phone's settings interface and set it to "Deep Noise Cancellation" mode to block out all external noise. When the user is in a supermarket, they need to reopen the settings interface, such as "Settings" - "Headphones" - "Noise Cancellation Mode" - "Natural Noise Cancellation", etc., which is cumbersome. Alternatively, another method for switching noise cancellation modes involves turning on the noise cancellation switch and simultaneously pressing the "Volume +" and "Volume -" buttons to cycle through the three noise cancellation modes. Pressing the "Volume +" and "Volume -" buttons once enters Natural Noise Cancellation mode. Pressing the "Volume +" and "Volume -" buttons a second time enters the ambient sound mode. Pressing the "Volume +" and "-" buttons a third time switches to "Deep Noise Cancellation" mode. This method of switching headphone noise cancellation modes via physical buttons requires users to press the physical buttons multiple times, which is inconvenient for users.

[0128] Step S34A: The device adjusts the noise reduction mode of the headphones based on the classification result of the first scene image.

[0129] In this embodiment, the mobile phone can recognize scene images and automatically adjust the noise cancellation mode of the headphones based on the classification results obtained from scene recognition, without requiring the user to manually set the noise cancellation mode. For example, different scenes have corresponding noise cancellation modes, and the mobile phone can adjust the noise cancellation mode according to the scene and the correspondence between the scene and the noise cancellation model. Different scenes and noise cancellation modes are shown in Table 1 below.

[0130] Table 1

[0131]

[0132] The correspondences between various noise reduction modes and scenes in Table 1 above are for illustrative purposes only and are not intended to impose limitations. The correspondences in Table 1 can be pre-configured by default. Alternatively, users can customize the correspondences between noise reduction modes and scenes according to their actual needs. For example, please refer to... Figures 11A-11C As shown, the phone displays a settings interface. The phone receives user selections (e.g., clicks) and determines the correspondence between each noise cancellation mode and a scene based on these selections. For example, in the Deep Noise Cancellation mode settings interface, if the user selects "Subway," "Airport," and "High-Speed ​​Rail," the phone establishes a correspondence between these three scenes and the Deep Noise Cancellation mode. Similarly, in the Everyday Noise Cancellation mode settings interface, if the user selects "Coffee Shop" and "Supermarket," the phone establishes a correspondence between Everyday Noise Cancellation mode and these two scenes. In the Monitoring Noise Cancellation mode settings interface, if the user selects "Office," the phone establishes a correspondence between Monitoring Noise Cancellation mode and "Office."

[0133] In another possible implementation, the mobile phone can automatically establish the correspondence between various noise cancellation modes and scenes based on the user's historical settings data for noise cancellation modes in different scenarios, using preset rules, statistical analysis, and / or statistical learning. The phone collects scene images of the user's current environment, uses a scene recognition model to identify the scene images, and obtains the recognition results, which indicate the user's current first scene (or environment). The phone queries historical setting data, which includes historical data on the correspondence between the user's first scene and each headphone noise cancellation mode. If the frequency of setting the correspondence between the first scene and the first noise cancellation mode is greater than a first threshold, the phone automatically establishes the correspondence between the first scene and the first noise cancellation mode. For example, taking "subway" as an example, the historical setting data is shown in Table 2 below.

[0134] Table 2

[0135]

[0136] As shown in Table 2 above, in historical settings data, in the subway scenario, the frequency of users setting "Deep Noise Cancellation Mode" (80%) is greater than the first threshold (e.g., the first threshold is 70%), while the frequency of users setting "Life Noise Cancellation Mode" (20%) is less than the first threshold. Therefore, the phone establishes a correspondence between "Subway" and "Deep Noise Cancellation Mode". In this implementation, personalized adjustment of the noise cancellation mode can be achieved without manual setting by the user. Optionally, users can also manually modify the correspondence automatically established by the phone for personalized configuration. For example, the correspondence between the first scenario and the first noise cancellation mode can be modified to a correspondence between the first scenario and the second noise cancellation mode. See, for example... Figure 12 As shown, the phone displays a settings interface, indicating a correspondence between "Subway" and "Deep Noise Cancellation Mode." "Deep Noise Cancellation Mode" is associated with a selection key. Responding to the user's selection key operation, the phone changes the correspondence between "Subway" and "Deep Noise Cancellation Mode" to "Subway" and "Lifestyle Noise Cancellation Mode." In this implementation, the phone can receive the user's selection operation, modify the automatically established correspondence between scenes and noise cancellation modes, and perform personalized configuration. This allows users to configure the correspondence between scenes and noise cancellation modes according to their environment and actual needs, improving the user experience.

[0137] In one application scenario of this application, a user is currently in a subway environment and listening to music through headphones. The user can turn on the phone's camera, or the phone's camera can be set to auto-start. The phone can capture a photo of the scene inside the subway using the camera. The phone can capture the image using either the front or rear camera; the specific method is not limited. For example, if the phone captures a scene image using the front camera, even if the scene image contains the user's "face," the scene recognition method in this embodiment can accurately identify the scene image as a first scene (such as a "subway" scene). Based on the first scene and the correspondence between the first scene and a first noise reduction mode (such as a deep noise reduction mode), the phone switches the headphone's noise reduction model to the first noise reduction mode (such as a deep noise reduction mode). In this application embodiment, the phone can perform scene recognition on the captured scene image and automatically adjust the headphone's noise reduction mode based on the scene recognition classification result, eliminating the need for the user to manually adjust the noise reduction mode, thus providing convenience.

[0138] For example, in another application scenario, if a user is currently using the video call function of an instant messaging app, the app has already accessed the camera, which is capturing images of the user's surroundings in real time. To reduce the computational load on the phone, the phone can acquire one frame of the scene image at intervals and then perform scene recognition on that image. For example, the duration of the time interval could be 10 minutes, 15 minutes, or 20 minutes. The duration of the time interval is based on the approximate time it takes for a user to move from one environment to another. Generally, users do not frequently change their environment in a short period. For example, a user moving from a "subway" to an "office," or from an "office" to a "supermarket," requires a time interval. For example, a 10-minute time interval is used. When a user is making a video call with another party using their phone, the phone's camera captures one frame of the scene image every 10 minutes. For example, at 10:20:01 on March 7, 2021, scene image A is captured. The phone classifies scene image A as "subway". Based on this classification, the phone adjusts its noise reduction mode to "deep noise reduction mode". The headphones then operate in deep noise reduction mode, making it almost impossible for the user to hear external noise, only the voice content of the other party in a video call. At 10:25:00 on March 7, 2021, the user exits the subway. At 10:30:01 on March 7, 2021, scene image B is captured. The phone classifies scene image B as "office". Based on this classification, the phone adjusts its noise reduction mode to "monitoring noise reduction mode". The headphones switch to monitoring noise reduction mode, blocking out ambient noise. The user cannot hear ambient noise, but can still hear colleagues greeting each other in the office environment and discussing issues, as well as the voice content of the other party in a video call. In this embodiment, the phone automatically adjusts the headphone noise reduction mode based on the scene recognition classification result, eliminating the need for the user to manually adjust the headphone noise reduction mode step by step, thus improving the user experience.

[0139] The second application scenario is explained below. System sounds include headphone, ringtone, call, and media audio. Users' system volume needs vary depending on their environment. For example, in noisy environments (such as subways or supermarkets), users need to turn up the system volume, such as ringtones and notification sounds, to avoid missing calls or messages, and call volume, to hear the other party clearly. However, in quieter environments (such as offices or libraries), users don't want the system volume to be too loud. Loud ringtones and notification sounds can disturb others, and loud call volumes may compromise privacy. In these quieter environments, users will lower the system volume. Thus, users may need to repeatedly adjust the system volume in different environments. Typically, to easily adjust the system volume, users in quiet environments simply mute the ringtone and notification sounds. While this avoids disturbing others, silencing the ringtone can also prevent users from promptly answering and replying to calls and messages.

[0140] Step S34B: The execution device adjusts its system volume based on the classification result of the first scene image.

[0141] In this embodiment, the mobile phone captures scene images, and can adaptively adjust the system volume based on the classification results of the scene images, eliminating the need for the user to frequently adjust the phone's system volume according to different environments. For an example, please refer to... Figure 13 As shown, the phone displays the system volume settings interface. This interface shows a progress bar for adjusting the volume value for each scenario, allowing users to adjust the volume by sliding the progress bar. Alternatively, in another implementation, the user doesn't need to set the volume value for different scenarios; the phone automatically configures the correspondence between different scenarios and system volume values ​​based on experience. The correspondence between different scenarios and system volume values ​​is shown in Table 3 below. The specific scenarios and their corresponding volume values ​​shown in Table 3 are merely illustrative examples and not limitations.

[0142] Table 3

[0143] Scene System volume subway 90 Airport 90 high-speed rail 80 coffee shop 50 supermarket 50 office 20

[0144] In one application scenario, a user is in a coffee shop environment. The user turns on their phone's camera, or the camera automatically activates. The phone captures an image C of the scene. The phone then performs scene recognition on image C, obtaining a classification result C (coffee shop scene). Based on this classification result C, the phone adjusts the system volume. For example, the phone adjusts the system volume to 50 based on the "coffee shop" scene. When a call comes in, the ringtone volume is set to 50, a low volume that won't disturb others while still allowing the user to hear the ringtone (or notification tone), preventing them from missing the call. When the user moves from the coffee shop to the subway, they are in the subway environment. The user turns on their phone's camera, or the camera automatically activates. The phone captures an image D of the scene. Based on image D, the phone recognizes the user's environment, obtaining a classification result D (subway scene). Based on this classification result D (subway scene), the phone adjusts the system volume to 90, ensuring the user can still hear the phone's system volume while on the subway.

[0145] For example, in another application scenario, if a user is currently using the video call function of an instant messaging app, the app has already accessed the camera, which is capturing images of the user's surroundings in real time. To reduce the computational load on the phone, the phone can acquire one frame of the scene image at intervals and then perform scene recognition on that image. For example, the duration of the interval could be 10 minutes, 15 minutes, or 20 minutes. The duration of the interval is based on the approximate time it takes for a user to move from one environment to another. When a user is making a video call, the phone's camera captures one frame of the scene image every 10 minutes. For example, if scene image C is captured at 10:20:01 on March 8, 2021, and the phone classifies scene image C as "subway," then the phone adjusts the headset volume to 90 based on this classification, increasing the volume in the headset so the user can clearly hear the sound. The user exited the subway at 10:25:00 on March 8, 2021. The phone captured scene image D at 10:30:01 on the same day. The phone classified scene image D as "office" and adjusted the headphone volume to 50 based on this classification. This reduced headphone volume ensured the user could hear the other person's voice, while maintaining a comfortable volume that wouldn't cause ear discomfort or leak the audio information.

[0146] In this embodiment, the mobile phone captures scene images of the user's environment through the camera, then identifies the scene images, and adjusts the system volume value adaptively based on the classification results of the scene images, i.e. the user's environment. This eliminates the need for the user to manually adjust the system volume repeatedly according to different environments, thus improving the user experience.

[0147] It should be understood that, in the above Figure 2B In the corresponding architecture, the user device (such as a mobile phone) receives the classification result of the first scene image to be identified from the execution device. The classification result is used to trigger the user device to adjust the noise cancellation mode of the headphones to the first noise cancellation mode. The scene indicated by the classification result corresponds to the first noise cancellation mode. For a detailed explanation of how the user device adjusts the noise cancellation mode of the headphones based on the classification result of the first scene image, please refer to the detailed explanation of step S34A above, which will not be repeated here.

[0148] Optionally, the user equipment (such as a mobile phone) receives the classification result of the first scene image to be identified from the execution device. The classification result is used to trigger the user equipment to adjust its system volume to the first volume value. The scene indicated by the classification result corresponds to the first volume value. For an explanation of how the user equipment adjusts its system volume based on the classification result of the first scene image, please refer to the detailed explanation of step S34B above, which will not be repeated here.

[0149] Compared to the above method embodiments, this application also provides an apparatus for applying the model training method. The model training method is applied to a model training apparatus, which may be the training device described in the above method embodiments, or the model training apparatus may be a processor in a training device, or the model training apparatus may be a chip system in a training device. Please refer to... Figure 14 As shown, this application provides an embodiment of a model training device 1400, which includes an acquisition module 1401 and a processing module 1402.

[0150] The acquisition module 1401 is used to acquire a first training dataset, which includes multiple first images;

[0151] The processing module 1402 is configured to: identify a first region in the first image using an object detection model, wherein the first region is an image region unrelated to scene recognition; perform masking processing on the first region to obtain a third image; acquire multiple sample object images generated by an image generation model, wherein the sample object images are images of objects unrelated to scene recognition; replace the masked first region in the third image with the multiple sample object images respectively to obtain multiple target images; train a first convolutional neural network using the dataset of the target images, and train a second convolutional neural network using the dataset of the third images to obtain a scene recognition model, wherein the scene recognition model includes the first convolutional neural network and the second convolutional neural network.

[0152] Optionally, the acquisition module 1401 can be replaced by a transceiver module. Optionally, the transceiver module is a transceiver. The transceiver has sending and / or receiving functions. Optionally, the transceiver can be replaced by a receiver and / or a transmitter.

[0153] Optionally, the transceiver module is a communication interface. Optionally, the communication interface is an input / output interface or a transceiver circuit. An input / output interface includes an input interface and an output interface. A transceiver circuit includes an input interface circuit and an output interface circuit.

[0154] Optionally, the processing module 1402 is a processor, which can be a general-purpose processor or a dedicated processor, etc. Optionally, the processor includes a transceiver unit for implementing receiving and transmitting functions. For example, this transceiver unit is a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing receiving and transmitting functions is deployed separately, or optionally, it is integrated together. The aforementioned transceiver circuit, interface, or interface circuit is used for reading and writing code or data, or for transmitting or relaying signals.

[0155] Furthermore, the acquisition module 1401 is used to perform the above... Figure 4 The corresponding step S10 in the example, Figure 7 The corresponding example is step S20. Processing module 1402 is used to execute the above. Figure 4 Steps S11-S14 in the corresponding example, Figure 7 The corresponding steps in the example are S20-S25.

[0156] Specifically, in one possible implementation, the processing module 1402 is further used for:

[0157] The first image is input into an image recognition model, and the image recognition model is used to obtain a first classification result of the first image and a heat map of the first image. The heat map is used to show the area where the target object is located. The image features of the target object are image features that are not related to scene recognition. The category indicated by the first classification result is a non-scene category or an incorrect scene category.

[0158] A masking process is applied to a second region in the first image, excluding the first region where the target object is located, to obtain a second image.

[0159] The first model is trained using a second training dataset to obtain the object detection model. The second training dataset includes multiple sample data, which includes input data and output data. The input data is the second image, and the output data is location coordinates, which are used to indicate the area where the target object is located.

[0160] In one possible implementation, the processing module 1402 is further configured to train the generative adversarial network (GAN) using the second image to obtain the image generation model.

[0161] In one possible implementation, the processing module 1402 is further specifically used for:

[0162] The first convolutional layer of the first convolutional neural network extracts the image features of the target image, and the second convolutional layer of the second convolutional neural network extracts the image features of the third image. The image features of the third image are then output to the first convolutional layer to be fused with the image features of the target image.

[0163] The output layer of the first convolutional neural network outputs the label of the first category based on the fused image features.

[0164] In one possible design, the functionality of processing module 1402 is implemented by a processing device, the functionality of which is partially or entirely implemented through software, hardware, or a combination thereof. Therefore, it can be understood that the above modules can be implemented through software, hardware, or a combination of both. In this case, the processing device includes a memory and a processor, wherein the memory stores computer programs, and the processor reads and executes the computer programs stored in the memory to perform the corresponding processes and / or steps in the above method embodiments. The processor includes, but is not limited to, one or more of a CPU, DSP, image signal processor, neural network processing unit (NPU), and microcontroller.

[0165] Optionally, the processing device includes only a processor. A memory for storing computer programs is located outside the processing device, and the processor is connected to the memory via circuitry / wires to read and execute the computer programs stored in the memory. Optionally, some or all of the functions of the processing device are implemented in hardware. In this case, the processing device includes input interface circuitry, logic circuitry, and output interface circuitry. Optionally, the processing device may be one or more chips, or one or more integrated circuits.

[0166] Optionally, the object detection model, image generation model, and scene recognition model can be neural network models, which can be embedded, integrated into, or run on a neural network processor (NPU).

[0167] Please see Figure 15As shown, for ease of understanding, the neural network processor 150 is briefly described. The neural network processor 150 is mounted as a coprocessor on the main processor, which may include, for example, a CPU, and is used to allocate tasks. The core of the neural network processor is the arithmetic circuit 1503, which is controlled by the controller 1504 to retrieve matrix data from memory and perform multiplication operations. In some implementations, the arithmetic circuit 1503 includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional systolic array. The arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0168] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0169] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via direct memory access controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0170] The bus interface unit (BIU) 1510 is used for interaction between the AXI bus and the DMAC and instruction fetch buffer 1509.

[0171] The bus interface unit 1510 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0172] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0173] The vector computation unit comprises over 1507 processing units, which further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computation in non-convolutional / FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0174] In some implementations, the vector computation unit 1507 stores the processed output vector into the unified memory 1506. For example, the vector computation unit 1507 can apply a nonlinear function to the output of the arithmetic circuit 1503, such as an accumulated vector of values, to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 1503, for example, for use in subsequent layers of a neural network. The instruction fetch buffer 1509, connected to the controller 1504, stores the instructions used by the controller 1504; the unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch buffer 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0175] Please see Figure 16 As shown, this application provides an electronic device 1600, which is a training device in the above method embodiments and is used to perform the functions of the training device in the above method embodiments. In this embodiment, the electronic device 1600 is described using a server as an example.

[0176] The server includes one or more central processing units (CPUs) 1622 (e.g., one or more processors) and memory 1632, and one or more storage media 1630 (e.g., one or more mass storage devices) for storing application programs 1642 or data 1644. The memory 1632 and storage media 1630 are either temporary or persistent storage. The program stored in the storage media 1630 includes one or more modules (not shown in the figure), each module comprising a series of instruction operations on the device. Furthermore, the CPU 1622 is configured to communicate with the storage media 1630 and execute the series of instruction operations stored in the storage media 1630 on the server.

[0177] Optionally, the server may also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input / output interfaces 1658, and / or one or more operating systems 1641.

[0178] Optionally, the server may also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input / output interfaces 1658, and / or one or more operating systems 1641.

[0179] Optionally, the central processing unit 1622 includes the above-mentioned Figure 15 The NPU shown.

[0180] Additionally, in an optional design, Figure 14 The function of the acquisition module 1401 is provided by Figure 16 The network interface 1650 is executed. Figure 14 The function of the processing module 1402 is as follows: Figure 16 The central processing unit 1622 in the system executes the commands.

[0181] This application also provides a scene recognition device used in the scene recognition method. The scene recognition device is used to perform the functions performed by the execution device in the above method embodiments. The scene recognition device may be the execution device in the above method embodiments, or it may be a processor in the execution device, or it may be a chip system in the execution device. Please refer to... Figure 17 As shown, this application provides an embodiment of a scene recognition device 1700, which includes an acquisition module 1701 and a processing module 1702. Optionally, the scene recognition device also includes a sending module 1703.

[0182] The acquisition module 1701 is used to acquire the first scene image to be recognized;

[0183] Processing module 1702 is used to detect a first region in the first scene image where an object unrelated to scene recognition is located using an object detection model;

[0184] The first region is masked to obtain the second scene image;

[0185] The first scene image is input into the first convolutional neural network in the scene recognition model, and the second scene image is input into the second convolutional neural network in the scene recognition model. The scene recognition model outputs a classification result. The first convolutional neural network is trained using a dataset of target images, and the second convolutional neural network is trained using a dataset of third images. The target image is obtained by replacing a first region in the third image with multiple sample object images generated by an image generation model. The third image is obtained by masking the first region in the first image after identifying a first region in the first image that is not related to scene recognition using an object detection model. The first image is an image from the training dataset.

[0186] Optionally, the object detection model, image generation model, and scene recognition model can be neural network models, which can be embedded, integrated, or run on the aforementioned models. Figure 15 The neural network processor (NPU) shown.

[0187] Optionally, the acquisition module 1701 can be replaced by a transceiver module. Optionally, the transceiver module is a transceiver. The transceiver has sending and / or receiving functions. Optionally, the transceiver can be replaced by a receiver and / or a transmitter.

[0188] Optionally, the transceiver module is a communication interface. Optionally, the communication interface is an input / output interface or a transceiver circuit. An input / output interface includes an input interface and an output interface. A transceiver circuit includes an input interface circuit and an output interface circuit.

[0189] Optionally, the processing module 1702 is a processor, which can be a general-purpose processor or a dedicated processor, etc. Optionally, the processor includes a transceiver unit for implementing receiving and transmitting functions. For example, this transceiver unit is a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing receiving and transmitting functions is deployed separately, or optionally, it is integrated together. The aforementioned transceiver circuit, interface, or interface circuit is used for reading and writing code or data, or for transmitting or relaying signals.

[0190] In one possible design, the functionality of processing module 1702 is implemented by a processing device, the functionality of which is partially or entirely implemented through software, hardware, or a combination thereof. Therefore, it can be understood that the above modules can be implemented through software, hardware, or a combination of both. In this case, the processing device includes a memory and a processor, wherein the memory stores computer programs, and the processor reads and executes the computer programs stored in the memory to perform the corresponding processes and / or steps in the above method embodiments. The processor includes, but is not limited to, one or more of a CPU, DSP, image signal processor, neural network processing unit (NPU), and microcontroller.

[0191] Optionally, the processing device includes only a processor. A memory for storing computer programs is located outside the processing device, and the processor is connected to the memory via circuitry / wires to read and execute the computer programs stored in the memory. Optionally, some or all of the functions of the processing device are implemented in hardware. In this case, the processing device includes input interface circuitry, logic circuitry, and output interface circuitry. Optionally, the processing device may be one or more chips, or one or more integrated circuits.

[0192] Furthermore, the acquisition module 1701 is used to execute the methods described in the above embodiments. Figure 10 The corresponding example is step S30. Processing module 1702 is used to execute the above method embodiment. Figure 10 The corresponding steps S31-S33 in the example. Optionally, when the executing device is a terminal device, the processing module 1702 is also used to execute steps S34A and S34B.

[0193] Specifically, in an optional implementation, the processing module 1702 is further configured to: extract image features of the first scene image through the first convolutional layer of the first convolutional neural network, extract image features of the second scene image through the second convolutional layer of the second convolutional neural network, and output the image features of the second scene image to the first convolutional layer for fusion with the image features of the first scene image; and output the classification result based on the fused image features through the output layer of the first convolutional neural network.

[0194] Specifically, in one optional implementation, if the classification result indicates a first scenario, the first scenario corresponds to the first noise reduction mode of the headphones;

[0195] The processing module 1702 is further configured to adjust the noise reduction mode of the headphones to the first noise reduction mode based on the classification result;

[0196] or,

[0197] The sending module 1703 is used to send the classification result to the user equipment, and the classification result is used to trigger the user equipment to adjust the noise cancellation mode of the headphones to the first noise cancellation mode.

[0198] In an optional implementation, if the classification result indicates a first scenario, the first scenario corresponds to a first volume value;

[0199] The processing module 1702 is further configured to adjust the system volume of the execution device to the first volume value according to the classification result;

[0200] or,

[0201] The sending module 1703 is used to send the classification result to the user equipment, and the classification result is used to trigger the user equipment to adjust the system volume of the user equipment to the first volume value.

[0202] Optionally, the transmitting module 1703 can be replaced by a transceiver module. Optionally, the transceiver module is a transceiver. The transceiver has transmitting and / or receiving functions. Optionally, the transceiver can be replaced by a receiver and / or a transmitter.

[0203] Optionally, the transceiver module is a communication interface. Optionally, the communication interface is an input / output interface or a transceiver circuit. An input / output interface includes an input interface and an output interface. A transceiver circuit includes an input interface circuit and an output interface circuit.

[0204] In an optional implementation, the acquisition module 1701 is further configured to receive a first scene image to be identified sent by a user equipment; or, acquire the first scene image to be identified through a camera or image sensor.

[0205] Please see Figure 18 As shown in the embodiments of this application, another electronic device is also provided. This electronic device 1800 is used to perform the functions performed by the execution device in the above method embodiments. In this embodiment, the electronic device is described using a mobile phone as an example. The electronic device 1800 includes components such as a processor 1801, a memory 1802, an input unit 1803, a display unit 1804, a camera 1805, a communication unit 1806, and an audio circuit 1807. The memory 1802 can be used to store software programs and modules. The processor 1801 executes various functional applications and data processing of the device by running the software programs and modules stored in the memory 1802. The memory 1802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The processor 1801 may be such as... Figure 17The processing device mentioned in the corresponding embodiment. Optionally, the processor 1801 includes, but is not limited to, various types of processors, such as one or more of the previously mentioned CPU, DSP, image signal processor, neural network processor as shown in 15, and microcontroller.

[0206] Input unit 1803 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the device. Specifically, input unit 1803 may include touch panel 1831. Touch panel 1831, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1831).

[0207] Display unit 1804 can be used to display various image information. Display unit 1804 may include display panel 1841, which may optionally be configured as a liquid crystal display, organic light-emitting diode, or the like. In some embodiments, touch panel 1831 can be integrated with display panel 1841 to realize the input and output functions of the device.

[0208] Camera 1805 is used to capture images of the scene to be identified, or to capture scene images and send the captured scene images to the database.

[0209] Communication unit 1806 is used to establish a communication channel, enabling the electronic device to connect to a remote server through the communication channel and obtain object detection models and scene recognition models from the remote server. The communication unit 1806 may include communication modules such as a wireless local area network module, a Bluetooth module, and a baseband module, as well as corresponding radio frequency (RF) circuits for wireless local area network communication, Bluetooth communication, infrared communication, and / or cellular communication system communication. The communication module is used to control the communication of various components in the electronic device and can support direct memory access.

[0210] Optionally, the various communication modules in the communication unit 1806 generally appear in the form of integrated circuit chips and can be selectively combined, without necessarily including all communication modules and corresponding antenna groups. For example, the communication unit 1806 may only include a baseband chip, a radio frequency chip, and a corresponding antenna to provide communication functionality in a cellular communication system. Through the wireless communication connection established via the communication unit 1806, the electronic device can connect to a cellular network or the Internet.

[0211] Audio circuitry 1807, speaker 1808, and microphone 1809 provide an audio interface between the user and the mobile phone. Audio circuitry 1807 converts received audio data into electrical signals and transmits them to speaker 1808, where speaker 1808 converts them into sound signals for output. Microphone 1809 converts collected sound signals into electrical signals, which are then received by audio circuitry 1807, converted back into audio data, and processed by processor 1801 before being transmitted via communication unit 1806 to, for example, another mobile phone, or output to memory 1802 for further processing.

[0212] In this embodiment, the electronic device is connected to an external headset via wired or wireless means (e.g., via Bluetooth). The communication unit 1806 is used to send a scene image to be identified to the training device and receive the classification result of the scene image from the server. The processor 1801 is also used to adjust the noise reduction mode of the headset according to the classification result. Alternatively, the processor 1801 is also used to adjust the system volume value according to the classification result.

[0213] Optionally, the processor 1801 performs scene recognition on the scene image to be recognized to obtain a classification result. The processor 1801 adjusts the noise cancellation mode of the headphones based on the classification result. Alternatively, the processor 1801 is also used to adjust the system volume value based on the classification result.

[0214] This application provides a computer-readable medium for storing a computer program that, when run on a computer, causes the computer to perform the method executed by the training device in the above method embodiments; or, when run on a computer, causes the computer to perform the method executed by the execution device in the above method embodiments.

[0215] This application provides a chip including a processor and a communication interface, such as an input / output interface, pins, or circuitry. The processor is used to read instructions to execute the method performed by the training device in the above method embodiments; or, the processor is used to read instructions to execute the method performed by the execution device in the above method embodiments.

[0216] This application provides a computer program product that, when executed by a computer, implements the method executed by the training device in the above method embodiments; or, when executed by a computer, implements the method executed by the execution device in the above method embodiments.

[0217] Optionally, the processor mentioned in any of the above places is a general-purpose central processing unit (CPU), a microprocessor, or an application-specific integrated circuit (ASIC).

[0218] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0219] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, Applied to training equipment, including: Obtain a first training dataset, which includes multiple first images; The object detection model is used to identify a first region in the first image, where the first region is an image region unrelated to scene recognition. The first region is masked to obtain the third image; Acquire multiple sample object images generated by the image generation model, wherein the sample object images are images of objects unrelated to scene recognition; The multiple sample object images are respectively replaced in the first area covered by the mask in the third image to obtain multiple target images; A first convolutional neural network is trained using the dataset of the target images, and a second convolutional neural network is trained using the dataset of the third images to obtain a scene recognition model, wherein the scene recognition model includes the first convolutional neural network and the second convolutional neural network. Wherein, both the target image and the third image correspond to a label of the first category, and the step of training a first convolutional neural network using the dataset of the target image and training a second convolutional neural network using the dataset of the third image includes: The first convolutional layer of the first convolutional neural network extracts the image features of the target image, and the second convolutional layer of the second convolutional neural network extracts the image features of the third image. The image features of the third image are then output to the first convolutional layer to be fused with the image features of the target image. The output layer of the first convolutional neural network outputs the label of the first category based on the fused image features.

2. The method according to claim 1, characterized in that, The method further includes: The first image is input into an image recognition model, and the image recognition model is used to obtain a first classification result and a heat map of the first image. The heat map is used to show the area where the target object is located. The image features of the target object are image features that are not related to scene recognition. The category indicated by the first classification result is a non-scene category or an incorrect scene category. A masking process is applied to a second region in the first image, excluding the first region where the target object is located, to obtain a second image. The first model is trained using a second training dataset to obtain the object detection model. The second training dataset includes multiple sample data, which includes input data and output data. The input data is the second image, and the output data is location coordinates, which are used to indicate the area where the target object is located.

3. The method according to claim 2, characterized in that, The method further includes: The second image is used to train the generative adversarial network (GAN) to obtain the image generation model.

4. A scene recognition method, characterized in that, Applied to execution devices, including: Obtain the first scene image to be identified; The object detection model is used to detect the first region in the first scene image where objects unrelated to scene recognition are located. The first region is masked to obtain the second scene image; The first scene image is input into the first convolutional neural network in the scene recognition model, and the second scene image is input into the second convolutional neural network in the scene recognition model. The scene recognition model outputs a classification result. The first convolutional neural network is trained using a dataset of target images, and the second convolutional neural network is trained using a dataset of third images. The target image is obtained by replacing the first region in the third image with multiple sample object images generated by the image generation model. The third image is obtained by masking the first region in the first image after identifying the first region that is not related to scene recognition using an object detection model. The first image is an image from the training dataset. The step of inputting the first scene image into the first convolutional neural network of the scene recognition model, inputting the second scene image into the second convolutional neural network of the scene recognition model, and outputting a classification result using the scene recognition model includes: The first scene image is extracted by the first convolutional layer of the first convolutional neural network, and the second scene image is extracted by the second convolutional layer of the second convolutional neural network. The image features of the second scene image are then output to the first convolutional layer to be fused with the image features of the first scene image. The classification result is output by the output layer of the first convolutional neural network based on the fused image features.

5. The method according to claim 4, characterized in that, If the classification result indicates a first scenario, the first scenario corresponds to the first noise cancellation mode of the headphones; The execution device is a terminal device, and the execution device is connected to the earphone. The method further includes: Based on the classification results, the noise cancellation mode of the headphones is adjusted to the first noise cancellation mode; or, The execution device is a server, the user equipment is connected to the headset, and the method further includes: The classification result is sent to the user equipment, and the classification result is used to trigger the user equipment to adjust the noise cancellation mode of the headphones to the first noise cancellation mode.

6. The method according to claim 4, characterized in that, If the classification result indicates a first scenario, then the first scenario corresponds to the first volume value; The execution device is a terminal device, and the method further includes: Based on the classification result, the system volume of the execution device is adjusted to the first volume value; or, The execution device is a server, and the method further includes: The classification result is sent to the user equipment, and the classification result is used to trigger the user equipment to adjust the system volume of the user equipment to the first volume value.

7. The method according to any one of claims 4-6, characterized in that, The acquisition of the first scene image to be identified includes: Receive the first scene image to be identified sent by the user equipment; or, The first scene image to be identified is acquired through a camera or image sensor.

8. An electronic device, characterized in that, include: A processor coupled to a memory for storing a program or instructions which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1 to 3; or, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 4 to 7.

9. A computer program product, said computer program product comprising computer program code, characterized in that, When the computer program code is executed by a computer, it causes the computer to implement the method as described in any one of claims 1 to 3; or, when the computer program code is executed by a computer, it causes the computer to implement the method as described in any one of claims 4 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs or instructions that, when executed, cause a computer to perform the method as described in any one of claims 1 to 3; or, when executed, cause a computer to perform the method as described in any one of claims 4 to 7.

Images