Machine vision understanding method, apparatus and device for scene information, and storage medium

Abstract

Disclosed in the present application is a machine vision understanding method for scene information. The method comprises: acquiring a scene image, the scene image containing target information of all targets; performing semantic segmentation on the scene image, and performing segmentation on targets of same types, so as to define different targets of each type as a plurality of independent instances; identifying each scene target, so as to acquire category information, state information and coordinate information of each target, and acquiring associations between the targets; on the basis of the category information, the coordinate information and the associations of the targets, constructing a knowledge graph of all the targets in the scene; and constructing a graph neural network on the basis of the knowledge graph, so as to generate a scene image state description. Further disclosed in the present invention are a machine vision understanding apparatus for scene information, a corresponding device and a storage medium. The machine vision understanding method for scene information provided by the present invention can provide more detailed monitoring scene information understanding processing results for visual monitoring systems, thus improving the efficiency of personnel in reviewing monitoring results.

Description

A method, apparatus, device, and storage medium for machine vision understanding of scene information. [Technical Field]

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, device, and storage medium for machine vision understanding of scene information. [Background Technology]

[0002] Industrial settings are large and contain numerous targets. Relying on monitoring systems for manual identification is time-consuming and prone to omissions, posing production safety hazards. Existing target recognition technologies primarily identify object types, lacking an understanding of the entire scene. They struggle to comprehensively assess all information within the scene, and qualitative and quantitative conclusions from multiple scenarios still require manual summarization, resulting in low monitoring efficiency and safety risks. [Summary of the Invention]

[0003] In view of at least one defect or improvement need of the prior art, the present invention provides a method, apparatus, device and storage medium for machine vision understanding of scene information, which will overcome at least one of the problems existing in the prior art.

[0004] To achieve the above objectives, according to a first aspect of the present invention, a machine vision understanding method for scene information is provided, the method comprising:

[0005] Acquire a scene image, which contains target information for all targets;

[0006] Semantic segmentation is performed on the scene image to segment targets of the same type, and each type of different targets is divided into multiple independent individuals;

[0007] For each scene target, identify the category information, state information and coordinate information of each target; obtain the correlation between targets, including physical correlation, spatial correlation and category correlation;

[0008] Based on the category information, coordinate information and correlation of the targets, a knowledge graph of all targets in the scene is constructed;

[0009] A graph neural network is constructed based on the knowledge graph, and a scene image state description is generated based on the graph neural network.

[0010] Furthermore, the above-mentioned machine vision understanding method for scene information performs semantic segmentation on the scene image, specifically including:

[0011] The scene image is semantically segmented, and different categories of targets are represented by differential masks.

[0012] Furthermore, in the aforementioned machine vision understanding method for scene information, the step of identifying each scene target and obtaining the category information, state information, and coordinate information of each target specifically includes:

[0013] Identify all targets in the scene and obtain category information for all targets;

[0014] Identify the state of all targets in the scene and obtain the state information of all targets;

[0015] Locate all targets in the scene and obtain the coordinate information of all targets;

[0016] Align the type information, status information, and coordinate information of each target.

[0017] Furthermore, in the above-mentioned machine vision understanding method for scene information, a knowledge graph of all targets in the scene is constructed based on the target's category information, coordinate information, and correlation. Here, "vertices" are used to represent the target's category information, state information, and coordinate information, and "edges" are used to represent the relationships between different "vertices".

[0018] Furthermore, in the above-mentioned machine vision understanding method for scene information, the construction of a graph neural network based on the knowledge graph specifically includes:

[0019] Using all "vertices" as the input to the graph neural network, the input information includes the category information, state information, and coordinate information contained in the "vertices";

[0020] When the previous layer of the network is passed to the next layer, the connection method between the "vertices" is consistent with the connection method of the knowledge graph.

[0021] According to a second aspect of the present invention, a scene information machine vision understanding device is also provided, comprising:

[0022] An acquisition module is configured to acquire a scene image containing target information of all targets;

[0023] The segmentation module is configured to perform semantic segmentation on the scene image, segment targets of the same type, and divide different targets of each type into multiple independent individuals;

[0024] The association module is configured to identify each scene target, obtain the category information, state information and coordinate information of each target; and obtain the association between targets, including physical association, spatial association and category association.

[0025] The building module is configured to construct a knowledge graph of all targets in the scene based on the target's category information, coordinate information, and correlation.

[0026] A generation module is configured to construct a graph neural network based on the knowledge graph and generate scene image state descriptions based on the graph neural network.

[0027] According to a third aspect of the present invention, a scene information machine vision understanding device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the methods described above.

[0028] According to a fourth aspect of the invention, a storage medium is also provided, which stores a computer program executable by a scene information machine vision understanding device, which, when run on the scene information machine vision understanding device, causes the scene information machine vision understanding device to perform the steps of any of the methods described above.

[0029] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0030] This invention provides a machine vision understanding method for scene information. Based on machine vision technology and knowledge graph technology, it can accurately segment and recognize scene images, accurately describe the scene, and further detect management information in multiple targets in the image. It can provide more detailed monitoring scene information understanding and processing results for visual monitoring systems, and greatly improve the efficiency of personnel reviewing monitoring results. [Attached Image Description]

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a flowchart illustrating a machine vision understanding method for scene information provided in an embodiment of this application;

[0033] Figure 2 is a schematic diagram of a scene image provided in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of scene image semantic segmentation provided in an embodiment of this application;

[0035] Figure 4 is a schematic diagram of scene image instance segmentation provided in an embodiment of this application;

[0036] Figure 5 is a schematic diagram of scene image target recognition and localization provided in an embodiment of this application;

[0037] Figure 6 is a schematic diagram of scene image correlation detection provided in an embodiment of this application;

[0038] Figure 7 is a schematic diagram of knowledge graph construction provided in an embodiment of this application;

[0039] Figure 8 is a schematic diagram of graph neural network construction provided in an embodiment of this application;

[0040] Figure 9 is a schematic diagram of the structure of a scene information machine vision understanding device provided in an embodiment of this application.

Detailed Implementation Methods

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0042] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0043] Figure 1 is a flowchart illustrating a machine vision understanding method for scene information provided in an embodiment of this application. As shown in Figure 1, the machine vision understanding method for scene information provided in an embodiment of this application includes:

[0044] 101. Obtain a scene image, which contains target information for all targets.

[0045] Specifically, scene images can be acquired through monitoring systems, camera devices, or other means of image acquisition, and the scene contains complete target information.

[0046] This application uses a boiler system scenario as an example to describe the implementation scheme of this application. Figure 2 is a scenario image of the boiler system provided in this application embodiment. As shown in Figure 2, the boiler system includes equipment such as a boiler, valves, a steam turbine, a condenser, and a water pump.

[0047] 102. Perform semantic segmentation on the scene image, segment targets of the same type, and divide different targets of each type into multiple independent individuals.

[0048] Specifically, semantic segmentation is performed on the scene, and different categories of targets are represented by differential masks. As shown in Figure 3, the entire boiler system scene is semantically segmented, and the equipment in the scene is divided into different categories, including boiler, valve, steam turbine, condenser and water pump, etc., and the different categories are represented by differential masks.

[0049] Instance-level segmentation is performed on targets of the same type, dividing each type of target into multiple independent individuals, as shown in Figure 4. For two water pumps that belong to the same category of water pumps, they are divided into different individuals.

[0050] 103. Identify each scene target and obtain the category information, status information and coordinate information of each target; obtain the correlation between targets, including physical correlation, spatial correlation and category correlation.

[0051] Specifically, the process iterates through the targets in each scene and locates them. This includes identifying all targets in the scene, obtaining their category information, identifying the state of each target, obtaining the target's state (for a boiler system scene, this state could include information such as noise, vibration, and temperature), and locating all targets in the scene, obtaining their coordinate information. The type, state, and location information of the same target are aligned to ensure a one-to-one correspondence between the target and its category, state, and coordinate information.

[0052] As shown in Figure 5, by identifying and locating the target, we can obtain the target's category information, status information, and coordinate information. For example, one target category is a regulating valve, the status is vibration, and the coordinates are (X1, Y1, Z1). Another target type is a water pump, the status is abnormal noise, and the coordinates are (X2, Y2, Z2).

[0053] It detects whether there is a correlation between adjacent or similar targets, and obtains the physical correlation, spatial correlation and category correlation between targets.

[0054] As shown in Figure 6, there is a physical connection between the water pump and the valve, that is, there is a physical correlation between the water pump and the valve. The steam turbine and the condenser are coupled together and have a spatial correlation. The state of one valve can affect the state setting of the other valve, that is, there is a category correlation between the two valves.

[0055] 104. Based on the category information, coordinate information and correlation of the targets, construct a knowledge graph of all targets in the scene.

[0056] Specifically, based on the identification type, location coordinates, and correlation, a knowledge graph of multiple targets in the scene is constructed to represent the information of the scene.

[0057] As shown in Figure 7, v represents a "vertex," indicating the target and its category, state, and coordinate attributes, while e represents an "edge," indicating the relationship between "vertexes" and other "vertexes." Specifically, "vertex" v1 represents the boiler, v2 and v6 represent valves, v3 represents the steam turbine, v4 represents the condenser, and v5 and v7 represent the water pumps. "Edges" e1 represent the relationships between devices when the working fluid is in a high-energy state, e2 represent the relationships between devices when the working fluid is in a low-energy state, and e3 represent the operational influence relationships between similar devices.

[0058] 105. Construct a graph neural network based on the knowledge graph, and generate scene image state descriptions based on the graph neural network.

[0059] Specifically, all "vertices" in the knowledge graph are used as the input to the network. The input information is the vertex type, state, and coordinates. When the previous layer of the network passes to the next layer, the connection method between the "vertices" is consistent with the connection method in the knowledge graph.

[0060] As shown in Figure 8, the input layer is connected to the first graph neural network layer in the same way as the knowledge graph in Figure 7. The first graph neural network layer is connected to the next graph neural network layer in the same way, and so on. The number of graph neural network layers can be selected according to the actual situation to build the graph neural network. Finally, it is connected to the prediction layer to describe the current scene state through the graph neural network.

[0061] The current scene state can be described based on the actual situation or user habits. For example, it can be described in the form of running logs, or in the form of annotations or highlights on the image. It can also be described in the form of voice or other methods that can describe the scene state, and the scene image can be transformed into inference results.

[0062] This application provides a machine vision understanding method for scene information. Based on machine vision technology and knowledge graph technology, it can accurately segment and recognize scene images, accurately describe the scene, and further detect management information in multiple targets in the image. This can provide more detailed monitoring scene information understanding and processing results for visual monitoring systems, and greatly improve the efficiency of personnel reviewing monitoring results.

[0063] This application also provides a scene information machine vision understanding device, as shown in Figure 9, including:

[0064] An acquisition module is configured to acquire a scene image containing target information of all targets;

[0065] The segmentation module is configured to perform semantic segmentation on the scene image, segment targets of the same type, and divide different targets of each type into multiple independent individuals;

[0066] The association module is configured to identify each scene target, obtain the category information, state information and coordinate information of each target; and obtain the association between targets, including physical association, spatial association and category association.

[0067] The building module is configured to construct a knowledge graph of all targets in the scene based on the target's category information, coordinate information, and correlation.

[0068] A generation module is configured to construct a graph neural network based on the knowledge graph and generate scene image state descriptions based on the graph neural network.

[0069] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0071] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0076] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0077] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A machine vision understanding method for scene information, characterized in that, include: Acquire a scene image, which contains target information for all targets; Semantic segmentation is performed on the scene image to segment targets of the same type, and each type of different targets is divided into multiple independent individuals; For each scene target, identify the category information, state information and coordinate information of each target; obtain the correlation between targets, including physical correlation, spatial correlation and category correlation; Based on the category information, coordinate information and correlation of the targets, a knowledge graph of all targets in the scene is constructed; A graph neural network is constructed based on the knowledge graph, and a scene image state description is generated based on the graph neural network.

2. The scene information machine vision understanding method as described in claim 1, characterized in that, Semantic segmentation of the scene image specifically includes: The scene image is semantically segmented, and different categories of targets are represented by differential masks.

3. The scene information machine vision understanding method as described in claim 1, characterized in that, The process of identifying each scene target and obtaining its category, state, and coordinate information specifically includes: Identify all targets in the scene and obtain category information for all targets; Identify the state of all targets in the scene and obtain the state information of all targets; Locate all targets in the scene and obtain the coordinate information of all targets; Align the type information, status information, and coordinate information of each target.

4. The scene information machine vision understanding method as described in claim 1, characterized in that, Based on the target's category information, coordinate information, and correlation, a knowledge graph of all targets in the scene is constructed. Here, "vertices" are used to represent the target's category information, state information, and coordinate information, and "edges" are used to represent the relationships between different "vertices".

5. The scene information machine vision understanding method as described in claim 4, characterized in that, The construction of a graph neural network based on the knowledge graph specifically includes: Using all "vertices" as the input to the graph neural network, the input information includes the category information, state information, and coordinate information contained in the "vertices"; When the data is passed from the previous layer to the next layer, the connection method between "vertices" is consistent with the connection method of the knowledge graph.

6. A machine vision understanding device for scene information, characterized in that, include: An acquisition module is configured to acquire a scene image containing target information of all targets; The segmentation module is configured to perform semantic segmentation on the scene image, segment targets of the same type, and divide different targets of each type into multiple independent individuals; The association module is configured to identify each scene target, obtain the category information, state information and coordinate information of each target; and obtain the association between targets, including physical association, spatial association and category association. The building module is configured to construct a knowledge graph of all targets in the scene based on the target's category information, coordinate information, and correlation. A generation module is configured to construct a graph neural network based on the knowledge graph and generate scene image state descriptions based on the graph neural network.

7. A scene information machine vision understanding device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of the method according to any one of claims 1 to 5.

8. A storage medium, characterized in that, It stores a computer program that can be executed by a scene information machine vision understanding device. When the computer program is run on the scene information machine vision understanding device, it causes the scene information machine vision understanding device to perform the steps of the method according to any one of claims 1 to 5.

Images