A scene recognition method, system, storage medium and terminal device

By acquiring and fusing the features and historical recognition results of the image set, the scene recognition model is used to improve the recognition accuracy of the zebra crossing theoretical intersection, and solve the problem of inaccurate scene recognition in the prior art.

CN114445760BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210079705.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-24
Publication Date
2024-11-26
Estimated Expiration
2042-01-24

AI Technical Summary

Technical Problem

The prior art has the problem of inaccurate scene recognition due to the complexity of objects in road pictures during the zebra crossing theoretical crossing recognition process.

Method used

By obtaining the set of images to be identified and the historical image set, the preset scene recognition model is used to determine the initial scene recognition results, and combined with the historical scene recognition results, the feature extraction, attention mechanism and recognition module fusion can be used to finally determine the set of images to be identified. Scene recognition results.

Benefits of technology

It improves the accuracy of scene recognition, reduces misjudgment caused by image blur or occlusion, and improves the recognition accuracy and automation rate of zebra crossing theoretical intersection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114445760B_ABST
    Figure CN114445760B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a scene recognition method, system, storage medium and terminal device, which are applied to the field of information processing technology based on artificial intelligence. The scene recognition system first determines the initial scene recognition result of whether the scene represented by the image set to be recognized belongs to a specific scene or does not belong to a specific scene, and combines the historical scene recognition results of the historical image set to determine the final scene recognition result of the image set to be recognized. In this way, the scene recognition results obtained by the scene recognition system for the image set related to at least one object can be integrated, and the accuracy of the final scene recognition result obtained is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information processing technology based on artificial intelligence, and in particular to a scene recognition method, system, storage medium and terminal device. Background Art

[0002] Image processing technology has been applied to a variety of application scenarios, such as user identification, transportation, and unmanned driving. Scene detection and recognition is a common application in these application scenarios. Generally, when performing scene detection and recognition, the image features of the image to be recognized can be analyzed to identify the various objects in the image to be recognized, and then the scene represented by the image to be recognized can be determined in combination with the recognized objects.

[0003] For example, when image recognition is applied to the recognition of scenes such as "theoretical intersection limit of zebra crossings", each object in the road image (such as the road surface, zebra crossings, etc.) can be recognized, and then combined with the recognized objects to determine whether the road image represents the theoretical intersection limit of zebra crossings. However, in the existing process of "theoretical intersection limit of zebra crossings", the recognition of the scene often has the problem of inaccuracy due to the complexity of each object in the road image. Summary of the invention

[0004] The embodiments of the present invention provide a scene recognition method, system, storage medium and terminal device, which improve the accuracy of recognizing a specific scene.

[0005] An embodiment of the present invention provides a scene recognition method, including:

[0006] Obtain a set of images to be identified;

[0007] Determining an initial scene recognition result of whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene;

[0008] Acquire a historical image set and a historical scene recognition result corresponding to the historical image set, wherein the historical image set and the image set to be recognized are image sets based on at least one object, or image sets whose similarity between sets is within a preset range;

[0009] A final scene recognition result of the set of images to be recognized is determined according to the initial scene recognition result and the historical scene recognition results.

[0010] Another aspect of the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the scene recognition method as described in one aspect of the embodiment of the present invention.

[0011] Another aspect of the present invention provides a scene recognition system, including:

[0012] An image collection unit, used for obtaining a collection of images to be identified;

[0013] An initial result unit, used to determine an initial scene recognition result of whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene;

[0014] A historical image unit, used to obtain a historical image set and a historical scene recognition result corresponding to the historical image set, wherein the historical image set and the image set to be recognized are image sets based on at least one object, or image sets whose similarity between sets is within a preset range;

[0015] The final recognition unit is used to determine the final scene recognition result of the set of images to be recognized according to the initial scene recognition result and the historical scene recognition result.

[0016] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0017] The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the scene recognition method as described in one aspect of an embodiment of the present invention; the processor is used to implement each computer program in the multiple computer programs.

[0018] It can be seen that in the method of this embodiment, the scene recognition system first determines the initial scene recognition result of whether the scene represented by the image set to be recognized belongs to a specific scene or does not belong to a specific scene, and combines the historical scene recognition results of the historical image set to determine the final scene recognition result of the image set to be recognized. In this way, the scene recognition results obtained by the scene recognition system for the image set related to at least one object can be integrated, and the accuracy of the final scene recognition result obtained is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 is a schematic diagram of a scene recognition method provided by an embodiment of the present invention;

[0021] Figure 2 is a flow chart of a scene recognition method provided by an embodiment of the present invention;

[0022] Figure 3 is a flow chart of a method for determining an initial scene recognition result in one embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a scene recognition model called in one embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of extracting overall features of a set of images to be identified in one embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of a scene recognition method in an application embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of a scene recognition model called in an application embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of an attention module in an application embodiment of the present invention;

[0028] Fig. 9 is a schematic diagram of a distributed system to which a scene recognition method in another application embodiment of the present invention is applied;

[0029] Fig.10 is a schematic diagram of a block structure in another application embodiment of the present invention;

[0030] Fig.11 is a logical structure diagram of a scene recognition system provided by an embodiment of the present invention;

[0031] Fig.12 It is a schematic diagram of the logical structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] The embodiment of the present invention provides a scene recognition method, which mainly recognizes the scene represented by an image through an image processing method, such as Figure 1 As shown, the scene recognition system can implement scene recognition in the following way:

[0035] Acquire a set of images to be identified; determine an initial scene recognition result indicating whether the scene represented by the set of images to be identified belongs to a specific scene or does not belong to a specific scene; acquire a set of historical images (n historical image sets are taken as an example in the figure) and historical scene recognition results corresponding to the set of historical images, wherein the set of historical images and the set of images to be identified are image sets based on at least one object, or are image sets whose similarities between sets are within a preset range; determine a final scene recognition result of the set of images to be identified based on the initial scene recognition result and the historical scene recognition results.

[0036] In the specific implementation process, the scene recognition system can be applied to but not limited to the following terminal devices: computers, intelligent voice interaction devices, smart home appliances, car terminals, etc.

[0037] In this way, the scene recognition results obtained by the scene recognition system for a set of images related to at least one object can be integrated, and the final scene recognition result obtained has higher accuracy.

[0038] The embodiment of the present invention provides a scene recognition method, which is mainly performed by a scene recognition system. The flow chart is as follows: Figure 2 As shown, including:

[0039] Step 101: Obtain a set of images to be identified.

[0040] It can be understood that the scene recognition system can initiate the method of this embodiment according to a certain period. Specifically, the scene recognition system can collect multiple images at multiple consecutive time points to form a set of images to be recognized, so as to initiate the process of this embodiment, or the scene recognition system can collect a video and intercept multiple frames of images in the video to form a set of images to be recognized, so as to initiate the process of this embodiment. Therefore, the set of images to be recognized can include multiple images.

[0041] Step 102: determining an initial scene recognition result of whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene.

[0042] Specifically, the scene recognition system can determine whether the scene represented by the image set to be recognized is a specific scene based on a preset scene recognition model. Here, the preset scene recognition model is a machine learning model based on artificial intelligence, which can be trained in advance through a certain training method, and its operation logic is stored in the scene recognition system. In this way, when the process of this embodiment is initiated, the preset scene recognition model can be directly called, and the called scene recognition model extracts the features of the image set to be recognized, and determines the initial scene recognition result based on the extracted features, wherein the features of the image set to be recognized may include the features of each image in the image set to be recognized and the context features between each image, etc.

[0043] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0044] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0045] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0046] It should be noted that the specific scenarios here may include scenarios in any application. For example, in vehicle driving applications, the specific scenarios may include but are not limited to the following scenarios: theoretical intersection limits of zebra crossings, theoretical intersection limits of logical isolation belts, theoretical intersection limits of obstacles, and lane number recognition, etc.

[0047] The theoretical intersection limit of a zebra crossing refers to a situation where the zebra crossing covers the entire intersection, preventing vehicles from passing in a certain direction. Specifically, if the zebra crossing is close to the lane line or the physical separation zone, preventing vehicles from turning left or making a U-turn before or after the zebra crossing, this scenario is the theoretical intersection limit of a zebra crossing. If the zebra crossing is dotted before and after, or there is enough distance between the zebra crossing and the physical separation zone for vehicles to make a U-turn, this is not the theoretical intersection limit of a zebra crossing. The theoretical intersection limit of a logical separation zone refers to a situation where the logical separation zone covers the entire intersection, preventing vehicles from passing through the separation zone. The theoretical intersection limit of an obstacle refers to a scenario where there is no free area in front of the obstacle and vehicles cannot pass through.

[0048] Furthermore, in order to improve the accuracy of the initial scene recognition result, the scene recognition system may also adjust the initial scene recognition result obtained in step 102. Specifically, if the initial scene recognition result includes information that the scene represented by the image set to be recognized belongs to a specific scene, and the objects involved in the images in the image set to be recognized include a specific area, the initial scene recognition result is adjusted to: information that the scene represented by the image set to be recognized does not belong to a specific scene.

[0049] The specific area is a distinguishing area for determining a specific scene from a non-specific scene, that is, if the image in the image set to be identified does not include the specific area, the scene represented by the image set to be identified belongs to the specific scene, and if the image in the image set to be identified includes the specific area, the scene represented by the image set to be identified does not belong to the specific scene, that is, the non-specific scene. When determining whether the image in the image set to be identified includes the specific area, the scene recognition system can identify whether each image in the image set to be identified includes the specific area according to the target recognition method.

[0050] For example, if the specific scenario is a theoretical intersection of a zebra crossing, the specific area is specifically a passable area, which refers to an area where vehicles are allowed to pass. If there is a dotted line in front of the zebra crossing and the dotted area is large enough for the vehicle to turn around again, that is, the area of ​​the dotted area is greater than a preset value, then the road surface area where the dotted line is located is the passable area.

[0051] Step 103, obtaining a historical image set and a historical scene recognition result corresponding to the historical image set, wherein the historical image set and the image set to be recognized are image sets based on at least one object, or image sets whose similarity between sets is within a preset range.

[0052] Specifically, when acquiring a historical image set, the scene recognition system can obtain one or more historical image sets consistent with the positioning information based on the positioning information of each image in the image set to be recognized; or, based on the objects involved in each image in the image set to be recognized, obtain that the images in the historical image set and the images in the image set to be recognized include at least one identical object, such as the same road sign or the same building.

[0053] Alternatively, when obtaining a historical image set, it is necessary to first calculate the similarity between the historical image set and the image set to be identified. The similarity can describe the similarity of images between the two sets. Only when the similarity between the historical image set and the image set to be identified is within a preset range, such as greater than a preset value, can it be used as a historical image set.

[0054] Here, each historical image set and its corresponding historical scene recognition result are: the recognition image set and the determined scene function results obtained by the scene recognition system in the scene recognition process initiated before the current process and stored in the system. When the scene recognition system initiates the current scene recognition process, the historical image set and its historical scene recognition results can be directly extracted from the system.

[0055] Step 104 : determining a final scene recognition result of the set of images to be recognized based on the initial scene recognition result and the historical scene recognition results.

[0056] Specifically, the scene recognition system can first calculate the result fusion parameter based on the historical scene recognition results. In this way, when the initial scene recognition result includes information that the scene represented by the image set to be recognized belongs to a specific scene, if the result fusion parameter belongs to a first preset range, the final scene recognition result is determined to be that the scene represented by the image set to be recognized belongs to the specific scene; if the result fusion parameter exceeds the first preset range, the final scene recognition result is determined to be that the scene represented by the image set to be recognized does not belong to the specific scene.

[0057] Furthermore, in the case where the initial scene recognition result includes information that the scene represented by the set of images to be recognized does not belong to a specific scene, if the result fusion parameter is within a second preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized belongs to the specific scene; if the result fusion parameter exceeds the second preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized does not belong to the specific scene.

[0058] The above result fusion parameter is used to represent the overall situation of the historical scene recognition result. Specifically, the result fusion parameter can be the weighted sum value R of the historical scene recognition result, which can be specifically represented by the following formula 1:

[0059]

[0060] Among them, r k represents the historical scene recognition result. If the historical scene recognition result is that the scene represented by the historical image set belongs to a specific scene, then r k =1, if the historical scene recognition result is that the scene represented by the historical image set does not belong to a specific scene, then r k =0. k It represents the weight value corresponding to the historical scene recognition result. Its specific value is related to factors such as the confidence of the historical scene recognition result and the acquisition time of the historical image set.

[0061] In this case, the first preset range is specifically greater than a first preset value, and the second preset range is specifically greater than a second preset value, where the second preset value may be greater than the first preset value.

[0062] It can be seen that in the method of this embodiment, the scene recognition system first determines the initial scene recognition result of whether the scene represented by the image set to be recognized belongs to a specific scene or does not belong to a specific scene, and combines the historical scene recognition results of the historical image set to determine the final scene recognition result of the image set to be recognized. In this way, the scene recognition results obtained by the scene recognition system for the image set related to at least one object can be integrated, and the accuracy of the final scene recognition result obtained is higher.

[0063] In a specific embodiment, when the scene recognition system performs the above step 102, Figure 3 As shown, this can be achieved through the following steps:

[0064] Step 201, calling a preset scene recognition model, the preset scene recognition model includes a feature extraction module, an object feature module, an attention module and a recognition module.

[0065] Specific as Figure 4The scene recognition model shown includes a feature extraction module 10 for extracting the overall features of the image set to be recognized, and the overall features may include the features of each image in the image set to be recognized and the context features between each image; the object feature module 11 extracts the features of multiple objects in the image set to be recognized based on the overall features extracted by the feature extraction module 10; the attention module 12 fuses the overall features extracted by the object feature module 11 and the features of multiple objects extracted by the object feature module 11 according to the attention mechanism to obtain fused features; the recognition module 13 determines whether the scene represented by the image set to be recognized belongs to a specific scene or not based on the fused features obtained by the attention module 12.

[0066] Step 202: extracting overall features of the set of images to be identified by a feature extraction module, and extracting features of multiple objects involved in the images in the set of images to be identified by an object feature module based on the overall features.

[0067] Specifically, when extracting overall features, the feature extraction module can obtain high-resolution features of the image set to be identified on the main channel, and the main channel includes multiple layers of high-resolution sub-features; at the same time, the feature extraction module obtains low-resolution features of the image set to be identified on the sub-channel based on the first layer of high-resolution sub-features in the multiple layers of high-resolution sub-features; finally, the feature extraction module fuses the high-resolution features and the low-resolution features to obtain the overall features of the image set to be identified.

[0068] like Figure 5 As shown, the feature extraction module is mainly divided into a main channel and three sub-channels. The main channel includes multiple layers of high-resolution sub-features, and each sub-channel includes multiple layers of low-resolution sub-features. Some layers of high-resolution sub-features in the main channel are obtained based on the previous layer of high-resolution sub-features and the first layer of low-resolution sub-features in sub-channel 1, another layer of high-resolution sub-features is obtained based on the previous layer of high-resolution sub-features, the first layer of low-resolution sub-features in sub-channel 1, and the second layer of low-resolution sub-features in sub-channel 2, and another layer of high-resolution sub-features is obtained based on the previous layer of high-resolution sub-features, the first layer of low-resolution sub-features in sub-channel 1, the second layer of low-resolution sub-features in sub-channel 2, and the third layer of low-resolution sub-features in sub-channel 3.

[0069] The initial low-resolution sub-features in sub-channel 1 are extracted from the above-mentioned first layer of high-resolution sub-features, some layers of low-resolution sub-features are obtained based on the previous layer of low-resolution sub-features and the second layer of low-resolution sub-features in sub-channel 2, and another layer of low-resolution sub-features is obtained based on the previous layer of low-resolution sub-features, the second layer of low-resolution sub-features in sub-channel 2, and the third layer of low-resolution sub-features in sub-channel 3.

[0070] The initial resolution sub-features in sub-channel 2 are extracted from the above-mentioned first layer of low-resolution sub-features, and some layers of low-resolution sub-features are obtained based on the previous layer of low-resolution sub-features and the third layer of low-resolution sub-features in sub-channel 3; the initial resolution sub-features in sub-channel 3 are extracted from the above-mentioned second layer of low-resolution sub-features.

[0071] Step 203: the attention module fuses the overall features and features of multiple objects according to the attention mechanism to obtain fused features.

[0072] Specifically, the attention module can weight the overall features and the features of multiple objects respectively, and use the weighted sum as the fused feature. When the attention module weights the overall features and the features of multiple objects, it needs to set a larger weight value for the features of several objects related to a specific scene, so that the recognition module will pay special attention to the features of these related objects in the process of recognizing the image set to be recognized, and finally improve the recognition accuracy of the image set to be recognized.

[0073] Step 204 : The recognition module determines whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene based on the fused features.

[0074] Specifically, the recognition module can output the probability that the scene represented by the image set to be identified belongs to a specific scene based on the fused features. If the probability is greater than or equal to a preset value, the scene represented by the image set to be identified is a specific scene. If the probability is less than the preset value, the scene represented by the image set to be identified is a non-specific scene.

[0075] The scene recognition method of the present invention is described below with a specific application example. The scene recognition system in this embodiment is mainly used in a vehicle-mounted device, and the vehicle-mounted device specifically recognizes whether the scene represented by the image set to be recognized is a zebra crossing theoretical intersection. Figure 6 As shown, the method of this embodiment includes the following steps:

[0076] Step 301, the vehicle-mounted device is installed at the front end of the vehicle. As the vehicle travels, the vehicle-mounted device can collect images of the road in front of the vehicle. Specifically, the vehicle-mounted device can collect multiple images of the road in front of the vehicle at multiple consecutive time points to form a set of images to be identified.

[0077] In step 302, the vehicle-mounted device calls a preset scene recognition model and uses the scene recognition model to determine an initial scene recognition result indicating whether the scene represented by the set of images to be recognized belongs to a characteristic scene or does not belong to a specific scene.

[0078] Specifically, the structure of the scene recognition model can be as follows Figure 7As shown, the feature extraction module extracts the overall features of the image set to be identified, and then the object feature module extracts the features of multiple objects involved in the image set to be identified based on the overall features, including the features of zebra crossings, lane lines and passable areas; then the attention module fuses the overall features and the features of multiple objects according to the attention mechanism to obtain fused features, so that the attention module focuses on the features of the areas before and after the zebra crossing to obtain fused features, and finally the recognition module can determine the initial scene recognition result according to the fused features.

[0079] Among them, the feature extraction module can specifically use a convolutional network to extract the overall features, which can be specifically used as mentioned above Figure 5 The high-resolution network (HRNet) shown in the figure extracts high-resolution features of the image set to be identified through the main channel, and extracts low-resolution sub-features from the first layer of high-resolution sub-features in the main channel through the sub-channel, thereby obtaining low-resolution features of the image set to be identified, and then fuses the low-resolution features and high-resolution features to obtain the overall features of the image set to be identified. Among them, the main channel will combine the low-resolution sub-features of the sub-channel, and the sub-channel will also combine the high-resolution sub-features of the main channel. The channels exchange information with each other to achieve multi-scale fusion and feature extraction, so that the overall features can more accurately describe the features of the image set to be identified.

[0080] Furthermore, if Figure 8 As shown in FIG. 1 , the attention module can use a squeeze-and-excitation network (SENet), which first fuses the overall features of the image set to be identified and the features of multiple objects to obtain the first fused feature X, and performs dimensionality reduction processing on the first fused feature X to reduce the spatial dimension of the first fused feature X to obtain the reduced feature U; feature compression is performed on the reduced feature U in the spatial dimension, and each two-dimensional feature channel is compressed into a real number, which represents the global distribution of the response on the corresponding feature channel, and enables the layer close to the input to also obtain the global perception; then a weight is generated for each feature channel through the parameter W, wherein the parameter W is learned to explicitly model the correlation between feature channels; finally, the parameter W is weighted to the previous feature channel by channel through multiplication, and the original feature is recalibrated in the channel dimension, thereby forming the re-fused feature X1.

[0081] In step 303, the vehicle-mounted device determines whether to adjust the initial scene recognition result. Specifically, if the initial scene recognition result includes information that the scene represented by the image set to be recognized belongs to a specific scene, and the objects involved in the images in the image set to be recognized include a specific area, which is a passable area in this embodiment, then the initial scene recognition result needs to be adjusted, and step 304 is executed; otherwise, in other cases, the initial scene recognition result does not need to be adjusted, and the following step 306 is directly executed.

[0082] In step 304 , the vehicle-mounted device adjusts the initial scene recognition result to the information that the scene represented by the set of images to be recognized does not belong to a specific scene, and continues to execute the following step 306 .

[0083] In step 305, the vehicle-mounted device obtains a historical image set, so that the historical image set and the image set to be identified are image sets based on at least one object, or image sets whose similarity between sets is within a preset range, and then obtains the historical scene recognition result corresponding to the historical image set.

[0084] It should be noted that after executing the above step 301, the vehicle-mounted device may, in one case, execute step 305 while executing step 302, or, in other cases, execute step 305 after executing step 303.

[0085] Step 306 , the vehicle fuses the initial scene recognition result and the historical scene recognition result to obtain a final scene recognition result corresponding to the set of images to be recognized.

[0086] In practical applications, the accuracy of zebra crossing theoretical intersection limit recognition will have a direct impact on the vehicle user experience. For example, redundant intersection limits will cause detours, and missing intersection limits will cause violations. In the road automation business, the high cost of zebra crossing theoretical intersection limit production line (requiring 220 people to work) and low quality (less than 90%) are one of the important factors affecting overall automation. The method of this embodiment can effectively improve the accuracy and automation rate of zebra crossing theoretical intersection limits by identifying such scenarios based on the set of images to be identified. Specifically:

[0087] (1) For the image set to be identified, the features of multiple objects can be fused with the overall features, and the attention mechanism can be used for feature processing, so that the accuracy of the initial scene recognition results obtained can be improved.

[0088] (2) The initial scene recognition results are adjusted based on the judgment of the passable area, thereby further improving the accuracy of the initial scene recognition results.

[0089] (3) The final scene recognition result is obtained by fusing the initial scene recognition result with the historical scene recognition result, which can effectively reduce the misjudgment caused by the blur and occlusion of the images in the image set to be recognized.

[0090] The following is another specific application example to illustrate the scene recognition method in the present invention. The information processing system in the embodiment of the present invention is mainly a distributed system 100, which may include a client 300 and multiple nodes 200 (any form of computing device in the access network, such as a server, a user terminal), and the client 300 and the node 200 are connected through network communication.

[0091] Take the distributed system as the blockchain system as an example, see Fig. 9 This is an optional structural diagram of the distributed system 100 provided by the embodiment of the present invention applied to the blockchain system, which is formed by multiple nodes 200 (any form of computing devices in the access network, such as servers and user terminals) and clients 300, and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0092] See also Fig. 9 The functions of each node in the blockchain system shown include:

[0093] 1) Routing: a basic function of a node, used to support communication between nodes.

[0094] In addition to the routing function, the node can also have the following functions:

[0095] 2) Applications, which are deployed in the blockchain to implement specific businesses according to actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to the temporary block when they successfully verify the source and integrity of the record data.

[0096] For example, the business implemented by the application includes code for implementing a scene recognition function, which mainly includes:

[0097] Acquire a set of images to be identified; determine an initial scene recognition result indicating whether the scene represented by the set of images to be identified belongs to a specific scene or does not belong to a specific scene; acquire a set of historical images (n historical image sets are taken as an example in the figure) and historical scene recognition results corresponding to the set of historical images, wherein the set of historical images and the set of images to be identified are image sets based on at least one object, or are image sets whose similarities between sets are within a preset range; determine a final scene recognition result of the set of images to be identified based on the initial scene recognition result and the historical scene recognition results.

[0098] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0099] See also Fig.10 An optional schematic diagram of the block structure provided by the embodiment of the present invention, each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block, and each block is connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains relevant information for verifying the validity of its information (anti-counterfeiting) and generating the next block.

[0100] The present invention also provides a scene recognition system, the structure diagram of which is shown in FIG. Fig.11 As shown, it may specifically include:

[0101] The image collection unit 20 is used to obtain a set of images to be identified.

[0102] The initial result unit 21 is used to determine an initial scene recognition result of whether the scene represented by the to-be-recognized image set acquired by the image set unit 20 belongs to a specific scene or does not belong to a specific scene.

[0103] The initial result unit 21 is specifically used to call a preset scene recognition model, which includes a feature extraction module, an object feature module, an attention module and a recognition module; the feature extraction module extracts the overall features of the image set to be recognized, and the object feature module extracts the features of multiple objects involved in the images in the image set to be recognized based on the overall features; the attention module fuses the overall features and the features of multiple objects according to the attention mechanism to obtain a fused feature; the recognition module determines whether the scene represented by the image set to be recognized belongs to a specific scene or not based on the fused features.

[0104] Among them, when the initial result unit 21 extracts the overall features of the image set to be identified through the feature extraction module, the feature extraction module is specifically used to obtain the high-resolution features of the image set to be identified on the main channel, and the main channel includes multiple layers of high-resolution sub-features; the feature extraction module obtains the low-resolution features of the image set to be identified on the sub-channel according to the first layer of high-resolution sub-features in the multiple layers of high-resolution sub-features; the feature extraction module fuses the high-resolution features and the low-resolution features to obtain the overall features.

[0105] Furthermore, the initial result unit 21 is also used to adjust the initial scene recognition result to: information that the scene represented by the image set to be recognized does not belong to a specific scene if the initial scene recognition result includes information that the scene represented by the image set to be recognized belongs to a specific scene; and the specific area is a distinguishing area for determining a specific scene from a non-specific scene.

[0106] The historical image unit 22 is used to obtain a historical image set and a historical scene recognition result corresponding to the historical image set. The historical image set and the image set to be recognized are image sets based on at least one object, or image sets with similarities between sets within a preset range.

[0107] The final recognition unit 23 is used to determine the final scene recognition result of the set of images to be recognized according to the initial scene recognition result determined by the initial result unit 21 and the historical scene recognition result obtained by the historical image unit 22 .

[0108] The final recognition unit 23 is specifically used to calculate a result fusion parameter based on the historical scene recognition result; if the initial scene recognition result includes information that the scene represented by the set of images to be recognized belongs to a specific scene, and the result fusion parameter belongs to a first preset range, it is determined that the final scene recognition result is that the scene represented by the set of images to be recognized belongs to the specific scene; if the result fusion parameter exceeds the first preset range, it is determined that the final scene recognition result is that the scene represented by the set of images to be recognized does not belong to a specific scene.

[0109] The final recognition unit 23 is further used to, if the initial scene recognition result includes information that the scene represented by the set of images to be recognized does not belong to a specific scene and the result fusion parameter is within a second preset range, determine that the final scene recognition result is that the scene represented by the set of images to be recognized belongs to a specific scene; if the result fusion parameter exceeds the second preset range, determine that the final scene recognition result is that the scene represented by the set of images to be recognized does not belong to a specific scene.

[0110] The final recognition unit 23, when calculating the result fusion parameter according to the initial scene recognition result and the historical scene recognition result, is specifically configured to use the result fusion parameter as a weighted sum value of the historical scene recognition results.

[0111] It can be seen that in the scene recognition system of this embodiment, the initial result unit 21 first determines the initial scene recognition result of whether the scene represented by the image set to be recognized belongs to a specific scene or does not belong to a specific scene, and the final recognition unit 23 combines the historical scene recognition results of the historical image set to determine the final scene recognition result of the image set to be recognized. In this way, the scene recognition results obtained by the scene recognition system for the image set related to at least one object can be integrated, and the accuracy of the final scene recognition result obtained is higher.

[0112] The embodiment of the present invention further provides a terminal device, the structural diagram of which is shown in FIG. Fig.12 As shown, the terminal device may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 30 (for example, one or more processors) and memory 31, and one or more storage media 32 (for example, one or more mass storage devices) storing application programs 321 or data 322. Among them, the memory 31 and the storage medium 32 can be short-term storage or permanent storage. The program stored in the storage medium 32 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the terminal device. Furthermore, the central processing unit 30 can be configured to communicate with the storage medium 32 to execute a series of instruction operations in the storage medium 32 on the terminal device.

[0113] Specifically, the application 321 stored in the storage medium 32 includes a scene recognition application, and the program may include the image collection unit 20, the initial result unit 21, the historical image unit 22 and the final recognition unit 23 in the above-mentioned scene recognition system, which will not be described in detail here. Furthermore, the central processor 30 can be configured to communicate with the storage medium 32 and execute a series of operations corresponding to the scene recognition application stored in the storage medium 32 on the terminal device.

[0114] The terminal device may also include one or more power supplies 33, one or more wired or wireless network interfaces 34, one or more input and output interfaces 35, and / or one or more operating systems 323, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0115] The steps performed by the scene recognition system in the above method embodiment can be based on the Fig.12 The structure of the terminal device shown.

[0116] Furthermore, another aspect of the embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the scene recognition method executed by the above-mentioned scene recognition system.

[0117] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0118] The memory is used to store a plurality of computer programs, and the computer programs are used to be loaded by the processor and execute the scene recognition method executed by the above-mentioned scene recognition system; the processor is used to implement each computer program in the plurality of computer programs.

[0119] In addition, according to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the scene recognition method provided in the above various optional implementations.

[0120] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0121] The above is a detailed introduction to a scene recognition method, system, storage medium and terminal device provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A scene recognition method, characterized in that: include: Get a set of images to be identified; Determining an initial scene recognition result of whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene; Acquire a historical image set and a historical scene recognition result corresponding to the historical image set, wherein the historical image set and the image set to be recognized are image sets based on at least one object, or image sets whose similarity between sets is within a preset range; Determining a final scene recognition result of the set of images to be recognized based on the initial scene recognition result and the historical scene recognition results; The step of determining the initial scene recognition result of whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene specifically includes: Calling a preset scene recognition model, wherein the preset scene recognition model includes a feature extraction module, an object feature module, an attention module, and a recognition module; The feature extraction module extracts the overall features of the image set to be identified, and the object feature module extracts the features of multiple objects involved in the images in the image set to be identified based on the overall features; The attention module fuses the overall feature and the features of multiple objects according to the attention mechanism to obtain a fused feature; The recognition module determines whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene according to the fused features; Determining the final scene recognition result of the set of images to be recognized based on the initial scene recognition result and the historical scene recognition result specifically includes: Calculate result fusion parameters according to the historical scene recognition results; If the initial scene recognition result includes information that the scene represented by the set of images to be recognized belongs to a specific scene, and the result fusion parameter belongs to a first preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized belongs to a specific scene; if the result fusion parameter exceeds the first preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized does not belong to a specific scene.

2. The method according to claim 1, characterized in that The extracting the overall features of the set of images to be identified by the feature extraction module specifically includes: The feature extraction module obtains high-resolution features of the set of images to be identified on a main channel, wherein the main channel includes multiple layers of high-resolution sub-features; The feature extraction module obtains low-resolution features of the set of images to be identified based on the first layer of high-resolution sub-features in the multiple layers of high-resolution sub-features on the sub-channels; The feature extraction module fuses the high-resolution features and the low-resolution features to obtain the overall features.

3. The method according to claim 1, characterized in that If the initial scene recognition result includes information that the scene represented by the set of images to be recognized belongs to a specific scene, the method further includes: If the objects involved in the images in the set of images to be identified include a specific area, the initial scene recognition result is adjusted to: information indicating that the scene represented by the set of images to be identified does not belong to a specific scene; The specific area is a distinguishing area for determining a specific scene and a non-specific scene.

4. The method according to claim 1, characterized in that The method further comprises: If the initial scene recognition result includes information that the scene represented by the set of images to be recognized does not belong to a specific scene, and the result fusion parameter is within a second preset range, it is determined that the final scene recognition result is that the scene represented by the set of images to be recognized belongs to a specific scene; if the result fusion parameter exceeds the second preset range, it is determined that the final scene recognition result is that the scene represented by the set of images to be recognized does not belong to a specific scene.

5. The method according to claim 1, characterized in that The calculating result fusion parameters according to the initial scene recognition result and the historical scene recognition result specifically includes: The result fusion parameter is a weighted sum value of the historical scene recognition results.

6. A scene recognition system, characterized in that: include: An image collection unit, used for obtaining a collection of images to be identified; An initial result unit is used to determine an initial scene recognition result of whether the scene represented by the image set to be recognized belongs to a specific scene or does not belong to a specific scene; the initial result unit is specifically used to call a preset scene recognition model, and the preset scene recognition model includes a feature extraction module, an object feature module, an attention module and a recognition module; the feature extraction module extracts the overall features of the image set to be recognized, and the object feature module extracts the features of multiple objects involved in the images in the image set to be recognized according to the overall features; the attention module fuses the overall features and the features of multiple objects according to the attention mechanism to obtain a fused feature; The recognition module determines whether the scene represented by the set of images to be recognized belongs to a specific scene or does not belong to a specific scene according to the fused features; A historical image unit, used to obtain a historical image set and a historical scene recognition result corresponding to the historical image set, wherein the historical image set and the image set to be recognized are image sets based on at least one object, or image sets whose similarity between sets is within a preset range; A final recognition unit, used to determine a final scene recognition result of the set of images to be recognized based on the initial scene recognition result and the historical scene recognition result; The final recognition unit is specifically used to calculate the result fusion parameter according to the historical scene recognition result; If the initial scene recognition result includes information that the scene represented by the set of images to be recognized belongs to a specific scene, and the result fusion parameter belongs to a first preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized belongs to a specific scene; if the result fusion parameter exceeds the first preset range, the final scene recognition result is determined to be that the scene represented by the set of images to be recognized does not belong to a specific scene.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor and executing the scene recognition method according to any one of claims 1 to 5.

8. A terminal device, characterized in that: including a processor and a memory; The memory is used to store a plurality of computer programs, and the computer programs are used to be loaded by the processor and execute the scene recognition method according to any one of claims 1 to 5; the processor is used to implement each computer program in the plurality of computer programs.