Multi-visual behavior monitoring method and device in complex scene, equipment and storage medium

Abstract

The invention discloses a multi-visual behavior monitoring method and device in a complex scene, equipment and a storage medium, and relates to the technical field of machine vision, and the method comprises the steps: extracting monitoring image information at multiple different angles based on a target positioning signal, extracting image features in the monitoring image information at multiple different angles, and obtaining a target positioning signal; image features of the target positioning signal are classified to obtain a plurality of target recognition features and a plurality of scene features, the plurality of target recognition features and the plurality of scene features are segmented respectively, pixel coordinates and the interaction relation between the target recognition features and the scene features are extracted, and the target recognition features and the scene features are identified; and determining behavior characteristics of a recognition target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, matching the behavior characteristics with a preset abnormal behavior library, obtaining a behavior monitoring feedback result when the matching succeeds, and outputting abnormal warning information according to the behavior monitoring feedback result, thereby realizing remote supervision, avoiding missing detection, reducing the inspection amount, and improving the accuracy of the monitoring result. The inspection efficiency is improved and the personnel inspection frequency is reduced.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method, device, equipment and storage medium for multi-visual behavior monitoring in complex scenarios. Background Art

[0002] In thermal power plants, the safety of the fuel area is extremely important. Due to its special environment, the safety of the fuel area needs to be strictly managed. At present, in order to protect the working safety of the fuel area, special inspection personnel are usually arranged to patrol the plant area to inspect whether the behavior of each equipment and staff in the inspection area complies with the regulations. However, manual inspection requires a lot of manpower costs, and manual inspection cannot be carried out without blind spots at all times, which is prone to safety hazards, such as conveyor belt deviation and equipment overheating, which are not easy to detect abnormal behaviors. Therefore, the current monitoring methods in thermal power plants have the problems of being unable to monitor at all times and covering all areas, low monitoring efficiency and low accuracy.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a multi-visual behavior monitoring method, device, equipment and storage medium in complex scenarios, aiming to solve the technical problems of low efficiency and low accuracy in abnormal behavior monitoring in the prior art.

[0005] To achieve the above purpose, the present application proposes a multi-visual behavior monitoring method in a complex scene, which is applied to the fuel area in the plant area. The fuel area deploys multiple surveillance cameras so that any area in the fuel area is monitored by at least two surveillance cameras. The multi-visual behavior monitoring method in a complex scene includes:

[0006] Extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features;

[0007] Segmenting the plurality of target recognition features and the plurality of scene features respectively, extracting pixel coordinates, and the interaction relationship between the target recognition features and the scene features;

[0008] Determine the behavior feature of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior feature with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful;

[0009] Output abnormal warning information according to the behavior monitoring feedback result.

[0010] In one embodiment, the steps of extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features include:

[0011] Determine the spatial coordinates of the identified target according to the target positioning signal, and determine the monitoring image information of the target monitoring camera according to matching the spatial coordinates with the monitoring area of ​​the monitoring camera;

[0012] Performing attribute prediction on the monitoring image information based on a multi-layer perceptron to obtain the relationship between objects in the monitoring image information, and obtaining a feature map based on the relationship map between the objects;

[0013] Performing text decoding on the feature map to determine a region of interest in the feature map;

[0014] Obtaining image features in the monitoring image based on the region of interest;

[0015] Detecting the target positioning signal to determine the signal category of the target positioning signal;

[0016] A feature classification strategy is determined according to the signal category, and the image features are classified according to the feature classification strategy to obtain a plurality of target recognition features and a plurality of scene features.

[0017] In one embodiment, the step of obtaining image features in the surveillance image based on the region of interest includes:

[0018] Determining scene text annotation information and the region of interest in the text annotation information of the feature map;

[0019] Determining the scene graph type from the feature graph according to a preset ratio of the scene text annotation information to the text annotation information;

[0020] When the scene graph type is a spatial scene graph, a position region sequence of a selected region in the spatial scene graph at a current moment is generated, and a probability of each position region in the position region sequence being selected is determined, and an attribute relationship between the region of interest and the spatial scene graph is obtained according to the probability of each position region being selected and a preset distribution parameter, and an image feature in the monitoring image is obtained according to the attribute relationship;

[0021] When the scene graph type is a semantic scene graph, weight information is determined according to the position of each semantic scene in the semantic scene graph and the region of interest, attribute relationships among the semantic scenes in the semantic scene graph are determined according to the weight information, and image features in the surveillance image are obtained according to the attribute relationships.

[0022] In one embodiment, the steps of segmenting the plurality of target recognition features and the plurality of scene features, extracting pixel coordinates, and the interactive relationship between the target recognition features and the scene features include:

[0023] Determine location information of a plurality of target identification features and a plurality of scene features in the monitoring image information respectively;

[0024] Determine edge information of the target recognition feature and the scene feature according to the position information, segment a plurality of the target recognition features and a plurality of the scene features respectively according to the edge information, and extract pixel coordinates of the target recognition features and the scene features;

[0025] The interactive relationship between the target recognition feature and the scene feature is determined according to the pixel coordinates of the target recognition feature and the scene feature.

[0026] In one embodiment, the step of determining the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship includes:

[0027] Detecting the target positioning signal category;

[0028] When the target positioning signal category is transportation equipment, according to the determined change trajectory of the pixel coordinates, the behavior characteristics of the transportation equipment are obtained according to the interactive relationship between the change trajectory and the scene characteristics;

[0029] When the target positioning signal category is a staff member, the skeleton information of the staff member is determined according to the pixel coordinates, the posture of the staff member is determined according to the skeleton information, and the behavioral characteristics of the staff member are determined based on the posture of the staff member and the interaction relationship.

[0030] In one embodiment, before the step of extracting surveillance image information at multiple different angles based on the target positioning signal, the step further includes:

[0031] receiving a target positioning signal sent by a signal generator, and determining the spatial coordinates of the target positioning signal in a spatial coordinate system established by the fuel area;

[0032] The spatial coordinates are overlapped with the monitoring area distribution diagram of the monitoring camera of the fuel area, the monitoring area contains the monitoring camera information of the position where the target positioning signal is located, and the image extraction information is generated based on the monitoring camera information.

[0033] In one embodiment, the multi-visual behavior monitoring method in complex scenes further includes:

[0034] Perform smoke and flame detection on monitoring image information;

[0035] When there is a smoke feature or a flame feature in the monitoring image area, determining the pixel coordinates of the smoke feature or the flame feature;

[0036] The camera monitoring area is determined according to the monitoring image information, and hidden danger alarm information is generated according to the camera monitoring area and the pixel coordinates of the smoke feature or the flame feature.

[0037] In addition, to achieve the above purpose, the present application also proposes a multi-visual behavior monitoring device in a complex scene, and the multi-visual behavior monitoring device in a complex scene includes:

[0038] A feature positioning module, used to extract monitoring image information of multiple different angles based on the target positioning signal, and extract image features from the monitoring image information of multiple different angles, and classify the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features;

[0039] A feature segmentation module, used to segment the plurality of target recognition features and the plurality of scene features respectively, extract pixel coordinates, and the interactive relationship between the target recognition features and the scene features;

[0040] A behavior monitoring module is used to determine the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior characteristics with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful;

[0041] The abnormal warning module is used to output abnormal warning information according to the behavior monitoring feedback result.

[0042] In addition, to achieve the above-mentioned objectives, the present application also proposes a multi-visual behavior monitoring device in complex scenes, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-visual behavior monitoring method in complex scenes as described above.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the multi-visual behavior monitoring method in complex scenes as described above are implemented.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multi-visual behavior monitoring method in complex scenes as described above.

[0045] One or more technical solutions proposed in the present application have at least the following technical effects: extracting monitoring image information of multiple different angles based on a target positioning signal, and extracting image features from the monitoring image information of multiple different angles, classifying the image features according to the target positioning signal to obtain multiple target identification features and multiple scene features, segmenting the multiple target identification features and the multiple scene features respectively, extracting pixel coordinates and the interaction relationship between the target identification features and the scene features, determining the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and matching the behavior characteristics with a preset abnormal behavior library, and obtaining a behavior monitoring feedback result when the match is successful, outputting abnormal warning information according to the behavior monitoring feedback result, effectively realizing remote supervision, avoiding missed inspections and reducing patrol workload, directly improving patrol efficiency, and reducing the frequency of personnel patrols. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A flowchart of the first embodiment of the multi-visual behavior monitoring method in complex scenarios provided by the present application;

[0049] Figure 2 A schematic diagram of video surveillance provided for an embodiment of a multi-visual behavior monitoring method in a complex scene of the present application;

[0050] Figure 3 A schematic diagram of image elements provided for an embodiment of a multi-visual behavior monitoring method in a complex scene of the present application;

[0051] Figure 4 A schematic diagram of a spatial scene provided for an embodiment of a multi-visual behavior monitoring method in a complex scene of the present application;

[0052] Figure 5 A schematic diagram of a relationship detection model provided in an embodiment of a multi-visual behavior monitoring method in a complex scenario of the present application;

[0053] Figure 6 A schematic diagram of a staff skeleton provided for an embodiment of a multi-visual behavior monitoring method in a complex scene of the present application;

[0054] Figure 7 A flowchart diagram of the second embodiment of the multi-visual behavior monitoring method in complex scenarios of this application is provided;

[0055] Figure 8 This is a schematic diagram of the module structure of a multi-visual behavior monitoring device in a complex scene according to an embodiment of the present application;

[0056] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the multi-visual behavior monitoring method in complex scenarios in the embodiment of the present application.

[0057] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of the embodiment of the present application is: extracting monitoring image information of multiple different angles based on the target positioning signal, and extracting image features from the monitoring image information of multiple different angles, classifying the image features according to the target positioning signal, and obtaining multiple target recognition features and multiple scene features; segmenting the multiple target recognition features and the multiple scene features respectively, extracting pixel coordinates, and the interaction relationship between the target recognition features and the scene features; determining the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and matching the behavior characteristics with a preset abnormal behavior library, and obtaining a behavior monitoring feedback result when the match is successful; outputting abnormal warning information according to the behavior monitoring feedback result.

[0061] In this embodiment, for ease of description, the following description is made by taking a multi-visual behavior monitoring device for identifying complex scenes as the execution subject.

[0062] Due to the existing technology, the safety of the fuel area is extremely important in thermal power plants, especially in the fuel area. Due to its special environment, the safety of the fuel area needs to be strictly managed. At present, in order to protect the working safety of the fuel area, special inspection personnel are usually arranged to patrol the plant area to inspect whether the behavior of each equipment and staff in the inspection area complies with the regulations. However, manual inspection requires a lot of manpower costs, and manual inspection cannot be carried out without blind spots at all times, which is prone to safety hazards, such as conveyor belt deviation and equipment overheating, which are difficult to detect abnormal behaviors. Therefore, the current monitoring method in thermal power plants has the problem of being unable to monitor with full coverage at all times, low monitoring efficiency and low accuracy.

[0063] This application provides a solution that uses AI intelligent inspection to achieve full-time monitoring of key areas without blind spots. Operation and maintenance personnel can check the on-site situation without on-site inspection, which can effectively reduce the safety production risks caused by equipment operation or mistaken entry into risky areas during inspection. It can avoid economic losses such as equipment overheating and belt deviation causing unit load reduction or unit non-stop.

[0064] It can be seen from the above embodiments that the present application extracts monitoring image information of multiple different angles based on the target positioning signal, and extracts image features from the monitoring image information of multiple different angles, classifies the image features according to the target positioning signal, obtains multiple target identification features and multiple scene features, respectively segments the multiple target identification features and the multiple scene features, extracts pixel coordinates, and the interaction relationship between the target identification features and the scene features, determines the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and matches the behavior characteristics with a preset abnormal behavior library. When the match is successful, obtains a behavior monitoring feedback result, outputs abnormal warning information according to the behavior monitoring feedback result, effectively realizes remote supervision, avoids missed inspections and reduces patrol workload, directly improves patrol efficiency, and reduces the frequency of personnel patrols.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a multi-visual behavior monitoring device in complex scenes, etc. The following takes a multi-visual behavior monitoring device in complex scenes as an example to illustrate this embodiment and the following embodiments.

[0066] Based on this, the embodiment of the present application provides a multi-visual behavior monitoring method in a complex scene, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the multi-visual behavior monitoring method in complex scenarios of the present application.

[0067] In this embodiment, the multi-visual behavior monitoring method in complex scenes includes steps S10 to S40:

[0068] Step S10, extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, classifying the image features according to the target positioning signal, and obtaining multiple target recognition features and multiple scene features.

[0069] It should be noted that the multi-visual behavior monitoring method in complex scenarios is applied to the fuel area in the plant. The fuel area deploys multiple surveillance cameras so that any area in the fuel area is monitored by at least two surveillance cameras. Each camera in the fuel area is assigned exclusive identity information, which can be based on the MAC address of each camera or the communication IP address information as identification information. The camera layout in the fuel area can be deployed in combination with the environment of the fuel area, refer to Figure 2 , Figure 2 The diagram is a camera monitoring diagram. Any position in the fuel area is monitored by two or more cameras, that is, any position in the fuel area will appear in two or more monitoring screens.

[0070] It should be understood that the target positioning signal refers to a signal used to describe the target position. The objects that need to be monitored include the equipment that needs to be monitored in the fuel area, and the maintenance personnel in the area, such as the belts in the conveyor device in the coal transportation system in the fuel area, the staff who start or maintain the equipment, etc. The target positioning signal can be issued by a signal generator installed or deployed on the equipment or related staff. The positioning signal of the equipment and the positioning signal of the staff can be distinguished. For example, when encoding the signal, the positioning information can be described by 32-bit data encoding and transmitted as a positioning signal. In the data encoding, the first bit can be set. The first bit 0 is the equipment type positioning signal, and the first bit 1 is the staff type positioning signal. The target positioning signal can be received by a signal receiver, and the signal receiver is deployed in a specific area within the fuel area. For example, the 32-bit data encoding is 0100100110100001 00010101 10000010, where bits 0 to 7 are the device representation information, bit 0 is 1, which is the device type positioning information, bits 8 to 15, 16 to 23, and 24 to 31 are the device's spatial coordinates, indicating the x, y, and z coordinate values ​​of the location.

[0071] It can be understood that the image feature refers to the feature of the element type in the image, including the attribute information of each element in the image information, including the type and name of each element, refer to Figure 3 , Figure 3 It is a schematic diagram of image elements. For the belt shown in the figure, the attribute information of the belt element is object: belt, relationship: contact with conveyor and coal, attribute: pixel proportion 30.72%. Target recognition features and scene features refer to image features in the image. The element corresponding to the target positioning information is the recognition target, which is extracted as the target recognition feature, and the other elements are scene features.

[0072] In a specific implementation, after obtaining the target positioning signal, the monitoring image information of multiple cameras that can capture the position corresponding to the target positioning signal can be extracted based on the target positioning signal, wherein the monitoring images of these cameras are monitoring images of the position at different angles, and feature recognition is performed on these multi-angle monitoring image information respectively to obtain image features in the monitoring image, including what each part of the image is, such as Figure 3 The image features of transmission equipment, coal, belts, walls, stairs, etc. can be extracted from the image shown. Since the monitoring image is acquired based on the target positioning signal, the location of the target positioning signal is the monitoring or extraction subject in this monitoring behavior or image feature extraction process, and other image features are automatically classified as scene features. Based on this, it can be understood that if the monitoring image can overlap with multiple identification positioning signals, but the input identification positioning signal is used as the main body for identification in a single image feature extraction, all other image features are scene features, that is, in an image feature extraction process, there is only one identification subject, namely, the target identification feature. The same processing process is performed in the monitoring image at each angle, and the target identification features consistent with the number of monitoring screens can be obtained, which is at least 2, as well as multiple scene features.

[0073] In a feasible implementation manner, the steps of extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features include:

[0074] Determine the spatial coordinates of the identified target according to the target positioning signal, and determine the monitoring image information of the target monitoring camera according to matching the spatial coordinates with the monitoring area of ​​the monitoring camera;

[0075] Performing attribute prediction on the monitoring image information based on a multi-layer perceptron to obtain the relationship between objects in the monitoring image information, and obtaining a feature map based on the relationship map between the objects;

[0076] Performing text decoding on the feature map to determine a region of interest in the feature map;

[0077] Obtaining image features in the monitoring image based on the region of interest;

[0078] Detecting the target positioning signal to determine the signal category of the target positioning signal;

[0079] A feature classification strategy is determined according to the signal category, and the image features are classified according to the feature classification strategy to obtain a plurality of target recognition features and a plurality of scene features.

[0080] It should be noted that the spatial coordinate refers to the representation form of describing the position in the spatial coordinate system established based on the fuel area. The process of establishing the spatial coordinate system is not limited in this embodiment. Object relationship refers to the relationship between the identified subject and the scene, which can be simply classified into two relationships: separation and intersection. Figure 3 For example, we can know that the relationship between the belt, transmission equipment and coal is an intersection relationship, and the relationship between the belt, transmission equipment, coal and the wall is a separation relationship.

[0081] It should be understood that the region of interest refers to the area where the identification target is located. In this embodiment, the region of interest can determine the initial position of the region of interest based on the identification positioning signal, and the region of interest can be adjusted based on the identification target. The classification strategy refers to the method used to classify signal categories. The signal categories can include at least transportation equipment and staff, where different signal categories are classified in different ways.

[0082] In a specific implementation, a spatial coordinate system can be established in the fuel area to describe the location information in the space. After the spatial coordinate system is established, it can be integrated according to the camera deployment map of the fuel area and the effective monitoring range of each camera, and the spatial coordinates can be matched with the monitoring area range of all monitoring cameras to determine whether the monitoring area range includes the target positioning signal. When the match is successful, that is, when the monitoring area range includes the target positioning signal, the current camera can be determined as the target monitoring camera, and the monitoring image information of the target monitoring camera can be extracted.

[0083] After obtaining the monitoring image information, the pre-trained Faster R-CNN model can be used to detect objects, and the attributes of objects can be predicted by using a multi-layer perceptron. The object relationship prediction is performed through MOTIFNET to obtain a monitoring image data set, including three types of data: objects, relationships, and attributes. The data can be annotated with text, and the nouns in the text can be classified and predicted through named entity recognition. Among them, objects that need to be identified, such as "staff" and "belts", can be determined as main nouns, and other nouns such as "walls" and "containers" can be set as scene nouns. By predicting the attributes of monitoring image information with a multi-layer perceptron, the relationship between objects in the monitoring image can be obtained, and a feature map can be obtained based on the relationship between objects.

[0084] Determine the scene text annotation information and the region of interest in the text annotation information of the feature graph; determine the scene graph type of the feature graph according to the size relationship of the scene text annotation information to the preset proportion of the text annotation information; when the scene graph type is a spatial scene graph, generate a position region sequence of the selected region in the spatial scene graph at the current moment, and determine the probability of each position region in the position region sequence being selected, obtain the attribute relationship between the region of interest and the spatial scene graph according to the probability of each position region being selected and a preset distribution parameter, and obtain the image features in the monitoring image according to the attribute relationship; when the scene graph type is a semantic scene graph, determine weight information according to the position of each semantic scene in the semantic scene graph and the region of interest, determine the attribute relationship between each semantic scene in the semantic scene graph according to the weight information, and obtain the image features in the monitoring image according to the attribute relationship; detect the target positioning signal and determine the signal category of the target positioning signal; determine a feature classification strategy according to the signal category, classify the image features according to the feature classification strategy, and obtain multiple target recognition features and multiple scene features.

[0085] In a specific implementation, the text annotation information of the input monitoring image information Image can be detected to determine the ratio of scene nouns to all nouns in the detected image. If it is greater than the preset ratio of 70%, the current monitoring image information Image can be determined to be a spatial scene graph, otherwise it is a semantic scene graph. In order to increase the accuracy of scene recognition, a scene flag can be added to the scene dictionary, the spatial scene graph is set to 0, the semantic scene graph is set to 1, and a scene dictionary with the current monitoring image information Image and the scene flag is generated.

[0086] When the monitoring image information Image is a spatial scene graph, we can focus on the objects in the local area and set a position variable S at time t t, which means focusing on the area under the image at time t, is one-hot encoded, with a weight α t,i is the image area A at time t i The probability of being selected, and only one region of interest can be selected at a time, so the position region sequence [S t,1 ,...S t,i ], when region i is selected, the probability value is 1, otherwise it is 0, so the region of interest can be described as: S t To generate the position information that the attention mechanism needs to pay attention to when generating the t-th word, the parameter is α i The multivariate Bernoulli distribution is: p(S t,i =1|S j<t ,A)=α t,i , construct object nodes based on the space, add a relationship node and multiple attribute nodes between objects, such as Figure 4 As shown in the spatial scene, edges from spatial object nodes to corresponding attribute nodes and edges pointing to relationship nodes between objects are established.

[0087] When the monitoring image information Image is a semantic scene graph, a multi-scene attention mechanism is used to focus on the global information in the visual relationship unit, and different weights are calculated for different positions. t,i Represents image area A i The proportion of all selected areas at time t is calculated by i And the corresponding weight α t，i Weighted, the final attention result is obtained: like Figure 4 As shown in the semantic scene in the figure, the area information of the object in the image is added to the graph neural network as an object node, and multiple attribute nodes are added, among which Figure 4 The diagonal nodes represent the relationship between objects, the black nodes represent objects in space, and the white nodes are the corresponding attributes of the objects.

[0088] In a feasible implementation manner, before the step of extracting monitoring image information of multiple different angles based on the target positioning signal, the step further includes:

[0089] receiving a target positioning signal sent by a signal generator, and determining the spatial coordinates of the target positioning signal in a spatial coordinate system established by the fuel area;

[0090] The spatial coordinates are overlapped with the monitoring area distribution diagram of the monitoring camera of the fuel area, the monitoring area contains the monitoring camera information of the position where the target positioning signal is located, and the image extraction information is generated based on the monitoring camera information.

[0091] It should be noted that the signal generator refers to a device for emitting a positioning signal, which can be installed or worn on the equipment or staff being behavior-detected, and is used to mark the current location of the equipment and the location of the staff.

[0092] In a specific implementation, multiple signal receivers are deployed in the fuel area, which can obtain the positioning information of the signal generator, wherein the position information can be determined by the joint positioning of three or more signal receivers, and the positioning information is linked to the spatial coordinate system constructed according to the fuel area, and the position of the signal generator can be determined according to the data of the signal receiver, and then the position information of the detected object is obtained, and the spatial coordinates are obtained. The spatial coordinates are overlapped with the distribution diagram of the monitoring area of ​​the fuel area monitoring camera, and the monitoring camera information of the monitoring area containing the position picture of the target positioning signal is determined, and the extraction image extraction information is generated based on the camera information.

[0093] Step S20, segmenting the plurality of target recognition features and the plurality of scene features respectively, extracting pixel coordinates, and the interaction relationship between the target recognition features and the scene features.

[0094] It should be noted that the target recognition features and scene features are in the same surveillance image. Therefore, when determining the interactive relationship between the target recognition features and the scene features, they may need to be segmented to segment the surveillance image into independent target recognition features and scene features. Since the role of scene features is consistent in the actual processing process, they are segmented as a whole.

[0095] In a specific implementation, when multiple target recognition features and multiple scene features are segmented, it can be described as determining the position information of the multiple target recognition features and the multiple scene features in the monitoring image information respectively; determining the edge information of the target recognition features and the scene features according to the position information, segmenting the multiple target recognition features and the multiple scene features respectively according to the edge information, and extracting the pixel coordinates of the target recognition features and the scene features; determining the interactive relationship between the target recognition features and the scene features according to the pixel coordinates of the target recognition features and the scene features. Since the segmentation method of the monitoring images of multiple angles is the same, the segmentation process is explained by taking one of the angle images as an example. During the segmentation process, the monitoring image can be processed to obtain a data set, and image information and annotation information can be obtained. The target recognition features and the scene features are determined based on the image information and the annotation information. The edge recognition is performed based on the target recognition features to obtain the pixel coordinates of the target recognition features in the image. The specific implementation is to extract the contour information of the target recognition features according to the Roberts Cross operator, and the calculation formula is:

[0096] Gx=f(i,j)-f(i-1,j-1)

[0097] Gy=f(i-1,j)-f(i,j-1)

[0098]

[0099] Among them, Gx and Gy represent the gradients in the horizontal and vertical directions, and ||G(x,y)|| is the gradient amplitude at the point.

[0100] After obtaining the target recognition feature contour, the pixel coordinates of the target recognition feature can be extracted, which is a set data type. After obtaining the pixel coordinates, the target recognition feature can be separated from the scene feature to obtain the target recognition feature and the scene feature. Then, the relationship between the target recognition feature and the scene feature is detected. Figure 5 , Figure 5 For the relationship detection model, the data in the dataset are processed by the object detector, the perception multi-layer perception layer, the Softmax function, and the flag encoding determined by the scene dictionary is output to the relationship detector to obtain the visual relationship.

[0101] When the target positioning signal category is transportation equipment, it is necessary to perform dynamic detection on continuous picture frames. Specifically, a motion curve is created based on the contour features of the target recognition features, and the contours of the target recognition features in several adjacent picture frames are used to determine the motion curve. For example, taking the belt in the coal transportation system as an example, during the coal transportation process, belt deviation will cause serious accidents in coal transportation. Therefore, it is necessary to detect the operating trajectory of the belt to determine whether there is a risk of deviation. Therefore, the monitoring images at various angles can be processed to determine the horizontal and vertical motion trajectories during the belt transmission process. This embodiment takes horizontal transportation as an example for explanation, and the processing idea of ​​inclined transportation is the same as that of horizontal transportation. Specifically, by marking the pixel points of multiple continuous picture frames, it is possible to fit the motion trajectory of the belt for a special point on the belt based on the position coordinates of the special point, that is, to obtain a motion trajectory, which is described as:

[0102]

[0103] Among them, a, b, c are constants, and t is the current time.

[0104] At the same time, the border of the transmission device used for reference under the belt is fitted as the scene reference, and a motion trajectory is fitted in a general way. The distance between the two motion trajectories is used to determine whether the belt is offset. If the distance is constant or changes within a preset threshold, the belt can be considered normal. At this time, the interaction relationship between the belt and the scene is a maintenance relationship. If the distance exceeds the preset threshold, the interaction relationship between the belt and the scene is considered to be an offset relationship.

[0105] When the target positioning signal category is staff, the scene overlap ratio (Scene Overlap Ratio, SOR) can be used to determine the interaction relationship between the staff and the scene. Therefore, the scene overlap ratio of the current monitoring image can be determined as:

[0106] SOR=(Area ofScene∩Area ofTarget) / Area ofTarget

[0107] Among them, Area of ​​Scene is the scene area, Area of ​​Target is the target recognition feature area, and SOR can reflect the positional relationship between the current target recognition features and scene features.

[0108] At this time, only the SOR of the current monitoring image is obtained. In order to determine the interactive relationship between the target recognition features and the scene features, the SOR of each angle can be calculated to determine the interactive relationship between the target recognition features and the scene features as follows:

[0109]

[0110] Wherein, i is the i-th monitoring image information, and n is the number of angles of the current monitoring image.

[0111] When R>0, it is determined that there is an interaction between the current target recognition feature and the scene feature. When R=0, it is determined that there is no interaction between the current scene features. For example, when a person climbs over a transmission device, the detection result is R>0, so it can be determined that the current staff has dangerous behavior. If R=0, it is determined that the current staff has not climbed over the transmission device.

[0112] Step S30, determining the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and matching the behavior characteristics with a preset abnormal behavior library, and obtaining a behavior monitoring feedback result when the match is successful.

[0113] It should be noted that the preset abnormal behavior library refers to a collection of various abnormal behaviors detected in past surveillance image information and abnormal behavior descriptions determined by behavioral norms.

[0114] In a specific implementation, when determining the behavioral characteristics of the recognition target corresponding to the target positioning signal based on pixel coordinates and pixel relationships, the category of the target positioning signal can be detected. When the category of the target positioning signal is a transportation device, according to the determined change trajectory of the pixel coordinates, the behavioral characteristics of the transportation device are obtained based on the interaction relationship between the change trajectory and the scene characteristics; that is, when the first digit code of the target positioning signal is 0 and it is determined that the positioning signal is a transportation device, the category to which the transportation device belongs is further determined. For example, when it is currently recognized that the conveyor belt of the transportation device is present, the movement of the belt is fitted, and the movement trajectory of the belt and the reference trajectory in the scene can be determined according to the pixel coordinates. By combining the positional change relationship between the movement trajectory and the reference trajectory, the current behavioral characteristics of the belt are obtained, and the current behavioral characteristics are matched with the abnormal behavior library. If the match is successful, that is, when the current belt is offset, a behavioral monitoring feedback result is obtained.

[0115] When the category of the target positioning signal is a staff member, the skeletal information of the staff member is determined according to the pixel coordinates, the body posture of the staff member is determined according to the skeletal information, and the behavioral characteristics of the staff member are determined based on the interaction relationship between the body posture of the staff member and the interaction relationship. At this time, the action state of the staff member can be fitted through consecutive frame images, and the skeletal information of the staff member can be determined, which can be determined by the geometric parameters of the key points in the skeleton. As Figure 6 shown, Figure 6 is a schematic diagram of the staff skeleton. The elbow and knee angles of the staff can be set as key parameters, θ0 is the bending angle of the spine, θ1 and θ2 are the knee angles respectively, and θ3 and θ4 are the elbow angles respectively. If the following conditions are satisfied in sequence, it can be determined that the staff member has a翻越动作 (climbing over action) and a behavioral monitoring feedback result is generated.

[0116] a. θ1 or θ2 > b1, and θ3 or θ4 < b2;

[0117] b. θ1 or θ2 < c;

[0118] c. θ0 < d1 and θ1 or θ2 < d2.

[0119] Step S40, output an abnormal warning message according to the behavioral monitoring feedback result.

[0120] It should be noted that the abnormal warning message can be output in the form of an alarm, and at the same time, it can be output in the fuel area and also in the monitoring area to achieve multi-faceted warning behavior.

[0121] In the specific implementation, when the behavior monitoring feedback results are received, the corresponding form of abnormal warning can be made according to the content of the detection feedback results. Taking the alarm sound as an example, the abnormal behavior of transportation equipment and the abnormal behavior of staff can be divided into two types of alarms for output. At the same time, according to the urgency of the situation, the alarm coverage and alarm volume are adjusted to promptly remind relevant personnel to intervene.

[0122] The present embodiment provides a multi-visual behavior monitoring method in a complex scene, which extracts monitoring image information of multiple different angles based on a target positioning signal, and extracts image features from the monitoring image information of multiple different angles, classifies the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features, respectively segment the multiple target recognition features and the multiple scene features, extracts pixel coordinates, and the interaction relationship between the target recognition features and the scene features, determines the behavior features of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and matches the behavior features with a preset abnormal behavior library. When the match is successful, a behavior monitoring feedback result is obtained, and abnormal warning information is output according to the behavior monitoring feedback result, so as to effectively realize remote supervision, avoid missed detection and reduce patrol workload, directly improve patrol efficiency, and reduce the frequency of personnel patrols.

[0123] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 7 The multi-visual behavior monitoring method in complex scenes also includes steps S50 to S70:

[0124] Step S50: performing smoke detection and flame detection on the monitoring image information;

[0125] Step S60: when there is a smoke feature or a flame feature in the monitoring image area, determining the pixel coordinates of the smoke feature or the flame feature;

[0126] Step S70: determining a camera monitoring area according to the monitoring image information, and generating hidden danger alarm information according to the camera monitoring area and the pixel coordinates of the smoke feature or the flame feature.

[0127] In a specific implementation, when the monitoring camera obtains the monitoring image information, smoke detection and flame detection can be performed synchronously on the currently obtained monitoring image information, wherein the flame detection includes flame warning monitoring and flame occurrence monitoring, wherein the flame warning monitoring monitors whether there is a possibility of fire, and the flame occurrence monitoring monitors whether there is a flame or a fire point. When performing smoke detection, the monitoring image can be grayed, and the smoke and background can be separated at this time. Then the image is inverted to obtain the smoke information in the image. After the smoke information is detected, the pixel coordinates of the current smoke can be determined, and the camera monitoring area can be determined according to the monitoring image information, and the spatial coordinates of the specific smoke generation point can be determined according to the camera monitoring area and the pixel coordinates, and the hidden danger alarm information is generated. Similarly, when performing flame detection, the high temperature abnormal point can be determined according to the infrared monitoring image, and the flame characteristics can be determined, and the corresponding pixel coordinates can be determined, and the camera monitoring area can be determined according to the monitoring image information, and the spatial coordinates of the specific flame generation point can be determined according to the camera monitoring area and the pixel coordinates, and the hidden danger alarm information is generated.

[0128] In this embodiment, smoke detection and flame detection can be performed based on monitoring image information. While monitoring the behavior of transportation equipment or personnel, safety hazards in the fuel area can be detected, making up for the shortcoming of only being able to monitor behavioral characteristics and further improving monitoring capabilities in complex scenarios.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multi-visual behavior monitoring method in complex scenarios of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a multi-visual behavior monitoring device in complex scenes, please refer to Figure 8 , the multi-visual behavior monitoring device in complex scenes includes:

[0131] The feature positioning module 10 is used to extract the monitoring image information of multiple different angles based on the target positioning signal, and extract the image features in the monitoring image information of multiple different angles, and classify the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features;

[0132] A feature segmentation module 20, for segmenting the plurality of target recognition features and the plurality of scene features, respectively, and extracting pixel coordinates and the interaction relationship between the target recognition features and the scene features;

[0133] A behavior monitoring module 30 is used to determine the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior characteristics with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful;

[0134] The abnormal warning module 40 is used to output abnormal warning information according to the behavior monitoring feedback result.

[0135] In one embodiment, the feature positioning module 10 is also used to determine the spatial coordinates of the identified target based on the target positioning signal, and determine the monitoring image information of the target monitoring camera by matching the spatial coordinates with the monitoring area of ​​the monitoring camera; perform attribute prediction on the monitoring image information based on a multi-layer perceptron to obtain the relationship between each object in the monitoring image information, and obtain a feature map based on the relationship map of each object; perform text decoding on the feature map to determine the area of ​​interest in the feature map; obtain image features in the monitoring image based on the area of ​​interest; detect the target positioning signal to determine the signal category of the target positioning signal; determine a feature classification strategy based on the signal category, and classify the image features according to the feature classification strategy to obtain multiple target recognition features and multiple scene features.

[0136] In one embodiment, the feature positioning module 10 is further used to determine the scene text annotation information and the region of interest in the text annotation information of the feature map;

[0137] Determining the scene graph type from the feature graph according to a preset ratio of the scene text annotation information to the text annotation information;

[0138] When the scene graph type is a spatial scene graph, a position region sequence of a selected region in the spatial scene graph at a current moment is generated, and a probability of each position region in the position region sequence being selected is determined, and an attribute relationship between the region of interest and the spatial scene graph is obtained according to the probability of each position region being selected and a preset distribution parameter, and an image feature in the monitoring image is obtained according to the attribute relationship;

[0139] When the scene graph type is a semantic scene graph, weight information is determined according to the position of each semantic scene in the semantic scene graph and the region of interest, attribute relationships among the semantic scenes in the semantic scene graph are determined according to the weight information, and image features in the surveillance image are obtained according to the attribute relationships.

[0140] In one embodiment, the feature segmentation module 20 is further used to determine the position information of a plurality of target recognition features and a plurality of scene features in the monitoring image information respectively;

[0141] Determine edge information of the target recognition feature and the scene feature according to the position information, segment a plurality of the target recognition features and a plurality of the scene features respectively according to the edge information, and extract pixel coordinates of the target recognition features and the scene features;

[0142] The interactive relationship between the target recognition feature and the scene feature is determined according to the pixel coordinates of the target recognition feature and the scene feature.

[0143] In one embodiment, the behavior monitoring module 30 is further used to detect the category of the target positioning signal;

[0144] When the target positioning signal category is transportation equipment, according to the determined change trajectory of the pixel coordinates, the behavior characteristics of the transportation equipment are obtained according to the interactive relationship between the change trajectory and the scene characteristics;

[0145] When the target positioning signal category is a staff member, the skeleton information of the staff member is determined according to the pixel coordinates, the posture of the staff member is determined according to the skeleton information, and the behavioral characteristics of the staff member are determined based on the posture of the staff member and the interaction relationship.

[0146] In one embodiment, the feature positioning module 10 is also used to receive a target positioning signal emitted by a signal generator, and determine the spatial coordinates of the target positioning signal in a spatial coordinate system established with the fuel area; overlap the spatial coordinates with the monitoring area distribution diagram of the monitoring camera of the fuel area, determine the monitoring area contains the monitoring camera information of the image where the target positioning signal is located, and generate image extraction information based on the monitoring camera information.

[0147] The multi-visual behavior monitoring device in complex scenes includes:

[0148] The hidden danger detection module 50 is used to perform smoke detection and flame detection on the monitoring image information;

[0149] A position determination module 60, for determining the pixel coordinates of the smoke feature or the flame feature when there is a smoke feature or a flame feature in the monitoring image area;

[0150] The hidden danger warning module 70 is used to determine the camera monitoring area according to the monitoring image information, and generate hidden danger alarm information according to the camera monitoring area and the pixel coordinates of the smoke feature or the flame feature.

[0151] The multi-visual behavior monitoring device for complex scenes provided by the present application adopts the multi-visual behavior monitoring method for complex scenes in the above-mentioned embodiments, which can solve the technical problems of low efficiency and low accuracy in abnormal behavior monitoring in the prior art. Compared with the prior art, the beneficial effects of the multi-visual behavior monitoring device for complex scenes provided by the present application are the same as the beneficial effects of the multi-visual behavior monitoring method for complex scenes provided by the above-mentioned embodiments, and the other technical features of the multi-visual behavior monitoring device for complex scenes are the same as the features disclosed in the above-mentioned embodiment methods, which will not be repeated here.

[0152] The present application provides a multi-visual behavior monitoring device for complex scenes, and the multi-visual behavior monitoring device for complex scenes includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-visual behavior monitoring method for complex scenes in the above-mentioned embodiment one.

[0153] Reference below Fig. 9 , which shows a schematic diagram of the structure of a multi-visual behavior monitoring device in a complex scene suitable for implementing the embodiment of the present application. The multi-visual behavior monitoring device in a complex scene in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The multi-visual behavior monitoring device in the complex scene shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0154] like Fig. 9As shown, the multi-visual behavior monitoring device under complex scenes may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multi-visual behavior monitoring device under complex scenes are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multi-visual behavior monitoring device in a complex scene to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a multi-visual behavior monitoring device in a complex scene with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0155] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0156] The multi-visual behavior monitoring device for complex scenes provided by the present application adopts the multi-visual behavior monitoring method for complex scenes in the above-mentioned embodiment, which can solve the technical problems of low efficiency and low accuracy in abnormal behavior monitoring in the prior art. Compared with the prior art, the beneficial effects of the multi-visual behavior monitoring device for complex scenes provided by the present application are the same as the beneficial effects of the multi-visual behavior monitoring method for complex scenes provided by the above-mentioned embodiment, and the other technical features of the multi-visual behavior monitoring device for complex scenes are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0159] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the multi-visual behavior monitoring method in complex scenes in the above-mentioned embodiment.

[0160] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0161] The computer-readable storage medium may be included in the multi-visual behavior monitoring device for complex scenes; or may exist independently without being assembled into the multi-visual behavior monitoring device for complex scenes.

[0162] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the multi-visual behavior monitoring device in a complex scene, the multi-visual behavior monitoring device in a complex scene:

[0163] Extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features;

[0164] Segmenting the plurality of target recognition features and the plurality of scene features respectively, extracting pixel coordinates, and the interaction relationship between the target recognition features and the scene features;

[0165] Determine the behavior feature of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior feature with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful;

[0166] Output abnormal warning information according to the behavior monitoring feedback result.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0170] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the multi-visual behavior monitoring method in complex scenes, and can solve the technical problems of low efficiency and low accuracy in abnormal behavior monitoring in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multi-visual behavior monitoring method in complex scenes provided in the above embodiments, and will not be repeated here.

[0171] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-visual behavior monitoring method in complex scenes as described above.

[0172] The computer program product provided by the present application can solve the technical problems of low efficiency and low accuracy in abnormal behavior monitoring in the prior art. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multi-visual behavior monitoring method in complex scenes provided by the above embodiment, which will not be repeated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multi-visual behavior monitoring method in complex scenes, characterized in that: Applied to the fuel area within the plant, the fuel area deploys multiple surveillance cameras so that any area within the fuel area is monitored by at least two surveillance cameras. The multi-visual behavior monitoring method in complex scenes includes: Extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features; Segmenting the plurality of target recognition features and the plurality of scene features respectively, extracting pixel coordinates, and the interaction relationship between the target recognition features and the scene features; Determine the behavior feature of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior feature with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful; Output abnormal warning information according to the behavior monitoring feedback result.

2. The method according to claim 1, characterized in that The steps of extracting surveillance image information of multiple different angles based on the target positioning signal, extracting image features from the surveillance image information of multiple different angles, and classifying the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features include: Determine the spatial coordinates of the identified target according to the target positioning signal, and determine the monitoring image information of the target monitoring camera according to matching the spatial coordinates with the monitoring area of ​​the monitoring camera; Performing attribute prediction on the monitoring image information based on a multi-layer perceptron to obtain the relationship between objects in the monitoring image information, and obtaining a feature map based on the relationship map between the objects; Performing text decoding on the feature map to determine a region of interest in the feature map; Obtaining image features in the monitoring image based on the region of interest; Detecting the target positioning signal to determine the signal category of the target positioning signal; A feature classification strategy is determined according to the signal category, and the image features are classified according to the feature classification strategy to obtain a plurality of target recognition features and a plurality of scene features.

3. The method according to claim 2, characterized in that The step of obtaining the image features in the monitoring image based on the region of interest comprises: Determining scene text annotation information and the region of interest in the text annotation information of the feature map; Determining the scene graph type from the feature graph according to a preset ratio of the scene text annotation information to the text annotation information; When the scene graph type is a spatial scene graph, a position region sequence of a selected region in the spatial scene graph at a current moment is generated, and a probability of each position region in the position region sequence being selected is determined, and an attribute relationship between the region of interest and the spatial scene graph is obtained according to the probability of each position region being selected and a preset distribution parameter, and an image feature in the monitoring image is obtained according to the attribute relationship; When the scene graph type is a semantic scene graph, weight information is determined according to the position of each semantic scene in the semantic scene graph and the region of interest, attribute relationships among the semantic scenes in the semantic scene graph are determined according to the weight information, and image features in the surveillance image are obtained according to the attribute relationships.

4. The method according to claim 1, characterized in that The steps of segmenting the plurality of target recognition features and the plurality of scene features respectively, extracting pixel coordinates, and the interactive relationship between the target recognition features and the scene features include: Determine location information of a plurality of target identification features and a plurality of scene features in the monitoring image information respectively; Determine edge information of the target recognition feature and the scene feature according to the position information, segment a plurality of the target recognition features and a plurality of the scene features respectively according to the edge information, and extract pixel coordinates of the target recognition features and the scene features; The interactive relationship between the target recognition feature and the scene feature is determined according to the pixel coordinates of the target recognition feature and the scene feature.

5. The method according to claim 1, characterized in that The step of determining the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship comprises: Detecting the target positioning signal category; When the target positioning signal category is transportation equipment, according to the determined change trajectory of the pixel coordinates, the behavior characteristics of the transportation equipment are obtained according to the interactive relationship between the change trajectory and the scene characteristics; When the target positioning signal category is a staff member, the skeleton information of the staff member is determined according to the pixel coordinates, the posture of the staff member is determined according to the skeleton information, and the behavioral characteristics of the staff member are determined based on the posture of the staff member and the interaction relationship.

6. The method according to claim 1, characterized in that Before the step of extracting monitoring image information of multiple different angles based on the target positioning signal, the method further includes: receiving a target positioning signal sent by a signal generator, and determining the spatial coordinates of the target positioning signal in a spatial coordinate system established by the fuel area; The spatial coordinates are overlapped with the monitoring area distribution diagram of the monitoring camera of the fuel area, the monitoring area contains the monitoring camera information of the position where the target positioning signal is located, and the image extraction information is generated based on the monitoring camera information.

7. The method according to claim 1, characterized in that The multi-visual behavior monitoring method in complex scenes also includes: Perform smoke and flame detection on monitoring image information; When there is a smoke feature or a flame feature in the monitoring image area, determining the pixel coordinates of the smoke feature or the flame feature; The camera monitoring area is determined according to the monitoring image information, and hidden danger alarm information is generated according to the camera monitoring area and the pixel coordinates of the smoke feature or the flame feature.

8. A multi-visual behavior monitoring device in complex scenes, characterized in that: The device comprises: A feature positioning module, used to extract monitoring image information of multiple different angles based on the target positioning signal, and extract image features from the monitoring image information of multiple different angles, and classify the image features according to the target positioning signal to obtain multiple target recognition features and multiple scene features; A feature segmentation module, used to segment the plurality of target recognition features and the plurality of scene features respectively, extract pixel coordinates, and the interactive relationship between the target recognition features and the scene features; A behavior monitoring module is used to determine the behavior characteristics of the identified target corresponding to the target positioning signal according to the pixel coordinates and the interaction relationship, and match the behavior characteristics with a preset abnormal behavior library, and obtain a behavior monitoring feedback result when the match is successful; The abnormal warning module is used to output abnormal warning information according to the behavior monitoring feedback result.

9. A multi-visual behavior monitoring device in complex scenes, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multi-visual behavior monitoring method in a complex scene as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multi-visual behavior monitoring method in a complex scene as described in any one of claims 1 to 7 are implemented.

Images