Intelligent inspection method and system based on artificial intelligence

Abstract

The invention relates to the technical field of machine vision and artificial intelligence, and discloses an intelligent inspection method and system based on artificial intelligence, and the method comprises the steps: collecting conventional inspection information; first structured data is constructed according to the collected conventional inspection information, the first structured data comprises two structural dimensions, basic structural elements included in the first structural dimensions are objects, and the objects of the first structured data comprise subareas of a monitoring area, a camera unit and a conventional inspection event; basic structural elements included in the second structural dimension are time sequence units; the method comprises the following steps: analyzing first structured data to obtain more than one object subset; respectively analyzing each object subset to obtain a comprehensive inspection event; according to the invention, the comprehensive inspection event can be dynamically identified according to the actual condition by using a small amount of data processing amount.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision and artificial intelligence, and more specifically, to an intelligent inspection method and system based on artificial intelligence. Background Art

[0002] The inspection mode adopted by video surveillance is to identify inspection-type events based on video streams, and then issue warnings and locate abnormal events. However, it only identifies the video stream of a single camera unit or a combination of video streams of more than two fixed camera units. Although the video streams of more than two camera units can be combined to jointly identify pre-set events, it is an impossible task to jointly identify the video streams of all camera units in the video surveillance system. Therefore, it is impossible to dynamically identify comprehensive inspection events according to actual conditions. Summary of the invention

[0003] The present invention provides an intelligent inspection method and system based on artificial intelligence to solve the technical problems in related technologies.

[0004] The present invention provides an intelligent inspection method based on artificial intelligence, comprising the following steps:

[0005] Step 1, collecting routine inspection information, which includes information about the camera unit of the monitoring system, partition information of the monitoring area, and routine inspection events identified by images collected by the camera unit in the first cycle;

[0006] Step 2, constructing the first structured data according to the collected routine inspection information, the first structured data includes two structural dimensions, the basic structural elements included in the first structural dimension are objects, and the objects of the first structured data include: partitions of the monitoring area, camera units, and routine inspection events; the basic structural elements included in the second structural dimension are time series units;

[0007] The correlations related to inspection tasks include:

[0008] Adjacency of subareas of the monitoring area;

[0009] The intersection or coverage of subareas of the monitoring area;

[0010] The camera unit is used to monitor the partitions of the surveillance area;

[0011] Routine inspection events occur in partitions of the monitored area.

[0012] Step 3, firstly analyze the first structured data to obtain more than one object subset, where one object subset includes at least two camera units, and after the conventional inspection information of the object subset is combined with the video streams of the camera units in the object subset, a comprehensive inspection event can be identified;

[0013] Step 4: For each object subset, analyze and obtain comprehensive inspection events.

[0014] Furthermore, the first cycle is an artificially defined cycle, which is used to divide the routine inspection information in time, for example, a routine inspection event of the first cycle is used as a data set.

[0015] Furthermore, the information of the camera unit includes:

[0016] The type and location of the camera unit and the divisions of the surveillance area that the camera unit can monitor.

[0017] The partition information of the monitoring area includes:

[0018] The ID, location, type and area of ​​the partition of the monitoring area.

[0019] Furthermore, the routine inspection events identified by the images captured by the camera unit are the events of routine analysis described above, and the analysis of routine inspection events is the result identified by the video stream of a single camera unit or the combination of video streams of more than two fixed camera units.

[0020] Furthermore, the method for constructing the first structured data includes:

[0021] extracting a first structured data object;

[0022] Extract specific features associated with the object, and divide the extracted specific features into multiple parts according to the collection period, and map one part to a time sequence unit. The order of the time sequence units is consistent with the time order of the collection period of the specific features;

[0023] Establishing a mapping between the extracted objects and their associated specific features;

[0024] For any two objects, if there is a correlation between the associated specific features related to the inspection task or there is a correlation between the objects related to the inspection task, an object relationship is established between the two objects.

[0025] Furthermore, a first model is constructed to complete step Step 3. The first model includes a first graph neural network layer, a first recurrent layer, and a first output layer. The first graph neural network layer inputs the first structured data, outputs the first hidden feature to the first recurrent layer, and the first recurrent layer outputs the second hidden feature to the first output layer. The first output layer includes multiple output steps, and an adjacency matrix is ​​output at each output step. The element in the i-th row and j-th column of the adjacency matrix indicates whether the i-th object and the j-th object belong to the same object subset. If so, the element value is 1, otherwise it is 0. Then, objects belonging to the same object subset are classified into an object subset, so an object subset is output at each output step.

[0026] Furthermore, a second model is constructed to analyze the object subset to obtain comprehensive inspection events. The second model includes a second graph neural network layer, a second loop layer and a second output layer. The second graph neural network layer inputs the first structured data, outputs the third hidden feature to the second loop layer, the second loop layer outputs the fourth hidden feature to the second output layer, and the second output layer outputs the type of comprehensive inspection events. The type of comprehensive inspection events includes no comprehensive inspection events, and the comprehensive inspection events are mapped to the object subset corresponding to the input second model.

[0027] Furthermore, the first model and the second model are jointly trained.

[0028] The present invention provides an artificial intelligence-based intelligent inspection system, which is configured with computer hardware capable of running computer-readable instructions and can execute the aforementioned artificial intelligence-based intelligent inspection method when running the computer-readable instructions.

[0029] The present invention provides a computer-readable storage medium for storing computer-readable instructions, which can execute the aforementioned intelligent inspection method based on artificial intelligence when the computer-readable instructions are read by a computer.

[0030] The beneficial effects of the present invention are as follows: the present invention performs a global analysis through conventional inspection information that does not include video streams, then determines a subset of objects that can identify comprehensive inspection events, and then imports a small amount of video stream information to identify comprehensive inspection events. It can use a small amount of data processing to dynamically identify comprehensive inspection events according to actual conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of an intelligent inspection method based on artificial intelligence of the present invention;

[0032] Figure 2 It is a diagram of the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0034] Routine inspection and analysis include: stall management: out-of-store operations, roving vendors, and business occupying the road; street management: random stacking of materials, business umbrellas, illegal advertising, and drying of garbage on the street; garbage management: packaged garbage, drying garbage, and overflowing garbage bins; vehicle management: illegal parking of non-motor vehicles and motor vehicles; timely detection and alarm positioning of the above problems to comprehensively improve the image of the community.

[0035] Specifically, in some embodiments, the following routine analysis is provided:

[0036] Street behavior analysis

[0037] This includes people running, people gathering, strenuous exercise, and people falling to the ground;

[0038] Street Event Analysis

[0039] Stall management: operating outside the store, roving vendors, and occupying the road;

[0040] Street management: random stacking of materials, umbrella business, illegal advertising, and drying clothes on the street;

[0041] Waste management: packing trash, drying trash, overflowing trash cans;

[0042] Vehicle management: non-motor vehicles and motor vehicles illegally parked are connected to the front-end video stream;

[0043] At least one embodiment of the present invention discloses an intelligent inspection method based on artificial intelligence, such as Figure 1 As shown, the following steps are included:

[0044] Step 1, collecting routine inspection information, which includes information of the camera unit of the monitoring system, partition information of the monitoring area (the monitoring area is divided into multiple partitions, and one partition can be monitored by at least one camera unit to collect image information), and routine inspection events recognized by the images collected by the camera unit in the first cycle;

[0045] The first cycle is a manually defined cycle, which is used to divide the routine inspection information in time, for example, a routine inspection event of the first cycle is taken as a data set;

[0046] The first cycle can be set to 30 minutes to 3 hours.

[0047] In some specific implementations of the present invention, the information of the camera unit includes:

[0048] The type and location of the camera unit and the divisions of the surveillance area that the camera unit can monitor.

[0049] The partition information of the monitoring area includes:

[0050] The ID of the partition of the monitoring area, the location of the partition, the type of partition (street, pavement, gate, school), and the area of ​​the partition.

[0051] The routine inspection events identified by the images captured by the camera unit are the events of the routine analysis mentioned above. The analysis of routine inspection events is the result of identification of the video stream of a single camera unit or the combination of video streams of more than two fixed camera units. Video stream recognition can be completed by combining a temporal neural network with a convolutional neural network.

[0052] Of course, generally, data preprocessing, including feature engineering, is performed while collecting data. The features described later are the results after data preprocessing, which is another form of data expression. However, the corresponding data is still identified by the meaning represented by the data in the embodiments.

[0053] Step 2, constructing the first structured data according to the collected routine inspection information, the first structured data includes two structural dimensions, the basic structural elements included in the first structural dimension are objects, and the objects of the first structured data include: partitions of the monitoring area, camera units, and routine inspection events; the basic structural elements included in the second structural dimension are time series units;

[0054] In some specific implementations of the present invention, the method for constructing the first structured data includes:

[0055] extracting a first structured data object;

[0056] Extract specific features associated with the object, and divide the extracted specific features into multiple parts according to the collection period, and map one part to a time sequence unit. The order of the time sequence units is consistent with the time order of the collection period of the specific features;

[0057] Establishing a mapping between the extracted objects and their associated specific features;

[0058] For any two objects, if there is a correlation between the associated specific features related to the inspection task or there is a correlation between the objects related to the inspection task, an object relationship is established between the two objects;

[0059] In some embodiments of the present invention, the associations related to the inspection tasks include:

[0060] Adjacency of subareas of the monitoring area;

[0061] The intersection or coverage of subareas of the monitoring area;

[0062] The camera unit is used to monitor the partitions of the surveillance area;

[0063] Routine inspection events occur in partitions of the monitored area.

[0064] The aforementioned objects: partitions of the monitoring area, camera units, and regular inspection events refer to the type of objects, not a single object. For example, there are multiple objects belonging to a partition of the monitoring area, and one object only refers to a partition of the monitoring area.

[0065] Step 3, firstly analyze the first structured data to obtain more than one object subset, where one object subset includes at least two camera units, and after the conventional inspection information of the object subset is combined with the video streams of the camera units in the object subset, a comprehensive inspection event can be identified;

[0066] Comprehensive inspection events refer to inspection events other than regular inspection events, such as: Cross-regional mass gatherings: In large public areas (such as squares or parks), people may gather from different directions and eventually converge. If each camera only focuses on its own local area, it may be difficult for the system to globally identify such large-scale gatherings;

[0067] Cross-regional collaborative criminal activities: Some organized criminal activities may occur simultaneously in different regions and are carried out by multiple people. If the system only focuses on local video streams, it may be difficult to identify these scattered but related criminal incidents;

[0068] Step 4: For each object subset, analyze and obtain comprehensive inspection events.

[0069] In some embodiments of the present invention, a first model is constructed to complete step Step3, the first model includes a first graph neural network layer, a first loop layer, and a first output layer, wherein the first graph neural network layer inputs the first structured data, outputs the first hidden feature to the first loop layer, the first loop layer outputs the second hidden feature to the first output layer, the first output layer includes multiple output steps, and outputs an adjacency matrix at each output step, the element in the i-th row and j-th column of the adjacency matrix indicates whether the i-th object and the j-th object belong to the same object subset, if so, the element value is 1, otherwise it is 0, and then the objects belonging to the same object subset are classified into an object subset, so an object subset is output at each output step;

[0070] In some embodiments of the present invention, a second model is constructed to analyze an object subset to obtain a comprehensive inspection event, the second model includes a second graph neural network layer, a second loop layer, and a second output layer, the second graph neural network layer inputs the first structured data, outputs the third hidden feature to the second loop layer, the second loop layer outputs the fourth hidden feature to the second output layer, the second output layer outputs the type of the comprehensive inspection event, the type of the comprehensive inspection event includes no comprehensive inspection event, and the comprehensive inspection event is mapped to the object subset corresponding to the input second model;

[0071] The first graph neural network layer and the second graph neural network layer both adopt a multi-layer structure, and the calculation formula of the lth layer is as follows:

[0072]

[0073] represents the object recognition feature of the vth object in the lth layer, represents the object recognition feature of the u-th object in the l-1th layer, N(v) and N(u) represent the sets of objects that have object relationships with object v and object u, respectively, || represents the cardinality of the set, Represents the plane recognition weight matrix of the lth layer of the first data structure recognition layer, l∈[1,E], E represents the total number of layers, and when l=1 x u Represents the feature associated with the u-th object, when l = E is equal to the first or third hidden feature of the object v (the first graph neural network layer outputs the first hidden feature, and the second graph neural network layer outputs the third hidden feature, but the trainable parameters of the first graph neural network layer and the second graph neural network layer are different), σ is the sigmoid function.

[0074] It should be noted that in the second graph neural network layer, the associated features of the camera unit object include the feature representation of the camera unit's video stream after feature engineering, while all global objects are input in the first graph neural network layer, so the associated features of the camera unit object do not include the camera unit's video stream.

[0075] That is, the routine inspection information of the object subset in Step 3 is combined with the video stream of the camera unit in the object subset, and the feature representation of the video stream of the camera unit after feature engineering processing is combined with the routine inspection information of the object subset as the object-associated feature input into the second graph neural network layer.

[0076] The feature engineering of video streams can be obtained by combining temporal neural networks with convolutional neural networks.

[0077] Both the first recurrent layer and the second recurrent layer adopt a temporal neural network, such as RNN, LSTM, or GRU.

[0078] One form of expression for the first and second recurrent layers is as follows:

[0079] ρ t =σ(Js t-1 +Ku t +l)

[0080] ζ t =σ(Ms t-1 +Nu t +m)

[0081]

[0082] ρ t and t : The activation vector of the reset gate and update gate at step t;

[0083] Candidate states generated in step t;

[0084] s t : The second or fourth hidden feature of the tth step (the first recurrent layer outputs the second hidden feature, and the second recurrent layer outputs the fourth hidden feature);

[0085] J, K, M, N, P, Q: 1st, 2nd, 3rd, 4th, 5th, 6th transformation matrices (trainable parameters);

[0086] l,m,n: 1st, 2nd, 3rd deviations (trainable parameters);

[0087] r∈G, G represents the set of all objects, t∈{1, 2, 3, ..., n}, t=1 when s t-1 =u t ;

[0088] represents the first hidden feature of the object v output when the t-th time series unit is input into the first graph network layer or the third hidden feature of the object v output when it is input into the second graph network layer, N(all) represents the set of all objects, and N(z) represents the set of all objects in the object subset input into the first graph neural network layer;

[0089] σ: Sigmoid activation function;

[0090] ψ: tanh activation function.

[0091] The first and second output layers are both fully connected layers.

[0092] In one embodiment of the present invention, because the output of the first model is part of the input of the second model, the first model and the second model can be jointly trained;

[0093] Alternatively, the first model is pre-trained and then jointly trained with the second model.

[0094] In one embodiment of the present invention, the first model and the second model are subjected to independent supervised training.

[0095] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

[0096] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 2 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to celestial body information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an intelligent inspection method based on artificial intelligence is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0097] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps in each of the above-mentioned embodiments of the intelligent inspection method based on artificial intelligence are implemented.

[0099] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in each of the above-mentioned embodiments of the intelligent inspection method based on artificial intelligence are implemented.

[0100] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

Claims

1. An intelligent inspection method based on artificial intelligence, characterized in that: The following steps are involved: Step 1, collecting routine inspection information, which includes information about the camera unit of the monitoring system, partition information of the monitoring area, and routine inspection events identified by images collected by the camera unit in the first cycle; Step 2, constructing the first structured data according to the collected routine inspection information, the first structured data includes two structural dimensions, the basic structural elements included in the first structural dimension are objects, and the objects of the first structured data include: partitions of the monitoring area, camera units, and routine inspection events; the basic structural elements included in the second structural dimension are time series units; Step 3, firstly analyze the first structured data to obtain more than one object subset, where one object subset includes at least two camera units, and after the conventional inspection information of the object subset is combined with the video streams of the camera units in the object subset, a comprehensive inspection event can be identified; Step 4: For each object subset, analyze and obtain comprehensive inspection events.

2. The intelligent inspection method based on artificial intelligence according to claim 1, characterized in that: The first cycle is a cycle defined manually and is used to divide the regular inspection information in time. For example, a regular inspection event of the first cycle is taken as a data set.

3. The intelligent inspection method based on artificial intelligence according to claim 1 is characterized in that: The camera unit information includes: The type and location of the camera unit and the divisions of the surveillance area that the camera unit can monitor. The partition information of the monitoring area includes: The ID, location, type and area of ​​the partition of the monitoring area.

4. The intelligent inspection method based on artificial intelligence according to claim 1 is characterized in that: The routine inspection events identified by the images captured by the camera unit are the events of routine analysis mentioned above. The analysis of routine inspection events is the result identified by the video stream of a single camera unit or the combination of video streams of more than two fixed camera units.

5. The intelligent inspection method based on artificial intelligence according to claim 1 is characterized in that: The method of constructing the first structured data includes: extracting a first structured data object; Extract specific features associated with the object, and divide the extracted specific features into multiple parts according to the collection period, and map one part to a time sequence unit. The order of the time sequence units is consistent with the time order of the collection period of the specific features; Establishing a mapping between the extracted objects and their associated specific features; For any two objects, if there is a correlation between the associated specific features related to the inspection task or there is a correlation between the objects related to the inspection task, an object relationship is established between the two objects.

6. The intelligent inspection method based on artificial intelligence according to claim 5 is characterized in that: The correlations related to inspection tasks include: Adjacency of subareas of the monitoring area; The intersection or coverage of subareas of the monitoring area; The camera unit is used to monitor the partitions of the surveillance area; Routine inspection events occur in partitions of the monitored area.

7. The intelligent inspection method based on artificial intelligence according to claim 1 is characterized in that: Construct a first model to complete step Step 3. The first model includes a first graph neural network layer, a first recurrent layer, and a first output layer. The first graph neural network layer inputs the first structured data, outputs the first hidden feature to the first recurrent layer, and the first recurrent layer outputs the second hidden feature to the first output layer. The first output layer contains multiple output steps, and outputs an adjacency matrix at each output step. The element in the i-th row and j-th column of the adjacency matrix indicates whether the i-th object and the j-th object belong to the same object subset. If so, the element value is 1, otherwise it is 0. Then, objects belonging to the same object subset are classified into an object subset, so an object subset is output at each output step.

8. The intelligent inspection method based on artificial intelligence according to claim 1, characterized in that: A second model is constructed to analyze the object subset to obtain comprehensive inspection events. The second model includes a second graph neural network layer, a second recurrent layer, and a second output layer. The second graph neural network layer inputs the first structured data, outputs the third hidden feature to the second recurrent layer, the second recurrent layer outputs the fourth hidden feature to the second output layer, and the second output layer outputs the type of comprehensive inspection events. The type of comprehensive inspection events includes no comprehensive inspection events, and the comprehensive inspection events are mapped to the object subset corresponding to the input second model.

9. An intelligent inspection system based on artificial intelligence, characterized in that: It is configured with computer hardware capable of running computer-readable instructions, and can execute the artificial intelligence-based intelligent inspection method as described in any one of claims 1-8 when running the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: It is used to store computer-readable instructions, and when the computer-readable instructions are read by a computer, the intelligent inspection method based on artificial intelligence as described in any one of claims 1 to 8 can be executed.

Images