An intelligent detection method and system for off-duty personnel based on machine vision

Through an intelligent detection system based on machine vision, combined with multiple sensors and deep learning algorithms, the problem of reduced accuracy among multiple monitors is solved, and high accuracy and real-time on-duty off-duty detection is achieved to reduce false alarms and enhance system reliability.

CN118015547BActive Publication Date: 2024-11-15CHENGDU QUDIAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410226046.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-28
Publication Date
2024-11-15
Estimated Expiration
2044-02-28

AI Technical Summary

Technical Problem

The accuracy of existing human video surveillance decreases when monitoring between multiple monitors, and there are problems such as high missed rate, slow response speed and poor reliability, making it difficult to effectively conduct on-duty off-duty testing.

Method used

Using an intelligent detection system based on machine vision, combining human sensing unit, image acquisition unit, voice acquisition unit, environmental monitoring module and image processing module, through a variety of sensor data and deep learning algorithms, we can judge whether there is a duty-free job in the target area in real time, including Infrared human body sensing components, photoelectric visibility sensors, ultrasonic visibility sensors, laser visibility sensors and voice acquisition units perform image preprocessing and feature extraction to judge the position, posture and behavior of employees.

Benefits of technology

Significantly reduce false alarm rates, improve detection accuracy and real-timeness, enhance the reliability and security of the system, and be able to complete the missing features through image correction models under low visibility conditions, ensuring the accuracy of off-job detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118015547B_ABST
    Figure CN118015547B_ABST
Patent Text Reader

Abstract

The present invention provides an intelligent detection method and system for off-duty on-duty personnel based on machine vision, and relates to the field of image processing. The system comprises: an image acquisition module, which is used to acquire image information and voice information of a target area; an environment monitoring module, which is used to acquire environment information of the target area; an image processing module, which is used to pre-process the image information of the target area based on the environment information of the target area to generate pre-processed image information; and an off-duty detection module, which is used to judge whether off-duty on-duty personnel occurs in the target area based on the pre-processed image information and voice information. The method has the advantages of automatically completing off-duty on-duty detection and improving the accuracy and real-time performance of detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for intelligently detecting off-duty personnel based on machine vision. Background Art

[0002] Some industries, such as mining, construction, and chemicals, will develop customized off-duty detection systems. The existing off-duty detection method is to install cameras in the workplace for monitoring, which requires additional personnel to monitor at all times. With the popularization and expansion of monitoring systems, the limitations of existing manual video monitoring to detect off-duty and off-duty are becoming more and more prominent. When the monitor observes multiple monitors at the same time, the monitoring accuracy will decrease as the number of monitoring videos increases. Since people need physiological activities such as eating and resting, this will also affect the reliability of the monitoring system. The existing manual monitoring system has problems such as high false alarm rate, slow response speed and poor reliability.

[0003] Therefore, it is necessary to provide an intelligent detection method and system for off-duty off-post based on machine vision, which can be used to automatically complete off-duty off-post detection and improve the accuracy and real-time performance of detection. Summary of the invention

[0004] One of the embodiments of the present specification provides an intelligent on-duty absence-from-post detection system based on machine vision, comprising: an image acquisition module, used to acquire image information and voice information of a target area; an environment monitoring module, used to acquire environmental information of the target area; an image processing module, used to pre-process the image information of the target area based on the environmental information of the target area to generate pre-processed image information; and an off-duty detection module, used to determine whether an on-duty absence-from-post occurs in the target area based on the pre-processed image information and voice information.

[0005] In some embodiments, the image acquisition module includes a human body sensing unit, an image acquisition unit and a voice acquisition unit, wherein the human body sensing unit includes a plurality of infrared human body sensing components arranged in the target area; the image acquisition module acquires image information and voice information of the target area, including: within a target detection cycle, judging whether there is a human body in the target area based on the output signals of the plurality of infrared human body sensing components; when it is judged that there is a human body in the target area, turning on the image acquisition unit to acquire image information of a marker in the target area at a preset posture; after acquiring the image information of the marker, determining an initial posture based on the output signals of the plurality of infrared human body sensing components, adjusting the image acquisition unit to the initial posture, and acquiring first image information of the target area; determining a target posture based on the first image information of the target area, adjusting the image acquisition unit to the target posture, and acquiring second image information of the target area; turning on the voice acquisition unit to acquire voice information of the target area.

[0006] In some embodiments, the environmental monitoring module includes a visibility monitoring unit and a meteorological information acquisition unit, wherein the visibility monitoring unit includes a photoelectric visibility sensor, an ultrasonic visibility sensor and a laser visibility sensor, and the meteorological information acquisition unit is used to acquire meteorological information of the target area.

[0007] In some embodiments, the image processing module preprocesses the image information of the target area based on the environmental information of the target area to generate preprocessed image information, including: determining the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area; when the visibility of the target area is less than a preset visibility threshold, correcting the second image information of the target area through an image correction model to generate the preprocessed image information.

[0008] In some embodiments, the image processing module determines the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area, including: determining a first visibility of the target area based on the image information of the marker and pre-stored image information; determining a second visibility of the target area based on the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor and the output signal of the laser visibility sensor; determining a third visibility of the target area based on the meteorological information of the target area; determining the visibility of the target area based on the first visibility, the second visibility and the third visibility.

[0009] In some embodiments, the image processing module determines the first visibility of the target area based on the image information of the marker and the pre-stored image information, including: acquiring a first region of interest image based on the image information of the marker; acquiring a second region of interest image based on the pre-stored image information; determining a first pixel RGB value matrix and a first edge pixel coordinate matrix based on the first region of interest image; determining a second pixel RGB value matrix and a second edge pixel coordinate matrix based on the second region of interest image; determining the first visibility of the target area based on the first pixel RGB value matrix, the first edge pixel coordinate matrix, the second pixel RGB value matrix and the second edge pixel coordinate matrix.

[0010] In some embodiments, the off-duty detection module determines whether the target area is off-duty based on the pre-processed image information and voice information, including: when the image acquisition module determines that there is no human body in the target area based on the output signals of the multiple infrared human body sensing components, recording the off-duty duration, and when the off-duty duration is greater than a preset time length threshold, determining that the target area is off-duty; the off-duty detection module determines whether the target area is off-duty based on the pre-processed image information and voice information, and also includes: extracting facial features based on the pre-processed image information, determining whether the preset on-duty personnel are on duty based on the facial features, and if it is determined that the preset on-duty personnel are not on duty, determining that the target area is off-duty; when it is determined that the preset on-duty personnel are on duty, extracting clothing features based on the pre-processed image information to determine whether the clothing features meet the preset clothing requirements, and when it is determined that the clothing features do not meet the preset clothing requirements, determining that the target area is off-duty.

[0011] In some embodiments, the off-post detection module is also used for: when it is determined that the dress features meet the preset dress requirements, based on the pre-processed image information corresponding to multiple first historical time points, extracting the first action feature sequence, the first expression feature sequence and the first sound feature sequence; based on the first action feature sequence, the first expression feature sequence and the first sound feature sequence, determining the first abnormal value; when the first abnormal value is greater than the preset abnormal value threshold, issuing a prompt message; based on the pre-processed image information corresponding to multiple second historical time points, extracting the second action feature sequence, the second expression feature sequence and the second sound feature sequence, wherein the second time point is after the prompt message is issued; based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, determining the second abnormal value, and when the second abnormal value is greater than the preset abnormal value threshold, issuing an alarm message.

[0012] In some embodiments, the off-post detection module determines the second outlier based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, including: determining the matching degree between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information based on the knowledge graph; determining the second outlier based on the matching degree between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information.

[0013] One of the embodiments of the present specification provides an intelligent detection method for off-duty personnel based on machine vision, including: acquiring image information and voice information of a target area; acquiring environmental information of the target area; preprocessing the image information of the target area based on the environmental information of the target area to generate preprocessed image information; and judging whether off-duty personnel have occurred in the target area based on the preprocessed image information and voice information.

[0014] Compared with the prior art, the intelligent detection method and system for off-duty personnel based on machine vision provided in this specification has at least the following beneficial effects:

[0015] 1. Reduce false alarm rate: By combining multiple sensor data and deep learning algorithms, normal and abnormal employee behaviors can be identified more accurately, significantly reducing the occurrence of false alarms. This means that operators will not be disturbed by unnecessary alarms, improving the reliability of the system;

[0016] 2. Multi-dimensional data analysis: Through the fusion of data collected by multiple devices such as human body sensing units, image acquisition units and voice acquisition units, the position, posture, sound and behavior of employees can be analyzed more comprehensively, which improves the detection ability of absenteeism events. When potential absenteeism events are found, alarm notifications can be issued in time, allowing operators to take immediate action, thus enhancing the safety of employees;

[0017] 3. When visibility is low, features of the second image information of the target area may be missing. By using the image correction model to make corrections based on the image information of the marker, pre-stored image information, visibility of the target area and the second image information of the target area, at least some of the missing image features can be supplemented, making subsequent off-duty detection more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0019] Figure 1 It is a module schematic diagram of a machine vision-based intelligent detection system for off-duty personnel according to some embodiments of this specification;

[0020] Figure 2 It is a flow chart of an intelligent detection method for off-duty personnel based on machine vision according to some embodiments of this specification;

[0021] Figure 3 is a schematic diagram of a process for obtaining image information of a target area according to some embodiments of this specification;

[0022] Figure 4 is a schematic diagram of a process for determining visibility of a target area according to some embodiments of this specification;

[0023] Figure 5 is a schematic diagram of a process of determining a first visibility of a target area according to some embodiments of this specification;

[0024] Figure 6 It is a schematic diagram of a process for determining whether a duty absence occurs in a target area according to some embodiments of this specification;

[0025] Figure 7 It is a schematic diagram of a relationship map shown in some embodiments of this specification. DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0027] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0028] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0029] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0030] Figure 1 is a module schematic diagram of a machine vision-based intelligent detection system for off-duty personnel according to some embodiments of this specification, such as Figure 1 As shown, an intelligent detection system for off-duty personnel based on machine vision may include an image acquisition module, an environment monitoring module, an image processing module and an off-duty detection module.

[0031] The image acquisition module can be used to acquire image information and voice information of the target area.

[0032] The target area may be the work and activity area of ​​the on-duty personnel.

[0033] In some embodiments, the image acquisition module includes a human body sensing unit, an image acquisition unit and a voice acquisition unit, wherein the human body sensing unit includes a plurality of infrared human body sensing components (e.g., infrared pyroelectric sensors, etc.) arranged in the target area. The image acquisition unit may include a posture adjustment component and a camera.

[0034] Figure 3 is a schematic diagram of a process of obtaining image information of a target area according to some embodiments of this specification, such as Figure 3 As shown, in some embodiments, the image acquisition module acquires image information and voice information of the target area, including:

[0035] In the target detection cycle, based on the output signals of the plurality of infrared human body sensing components, it is determined whether there is a human body in the target area. For example, when at least two infrared human body sensing components detect a human body, it is determined that there is a human body in the target area.

[0036] When it is determined that there is a human body in the target area, the image acquisition unit is turned on to acquire image information of a marker in the target area at a preset position, wherein the marker may be an object whose position in the target area does not change;

[0037] After acquiring the image information of the marker, an initial position is determined based on the output signals of the plurality of infrared human body sensing components, and the image acquisition unit is adjusted to the initial position to acquire first image information of the target area;

[0038] Based on the first image information of the target area, determine the target posture, adjust the image acquisition unit to the target posture, and acquire second image information of the target area;

[0039] Turn on the voice collection unit to obtain voice information in the target area.

[0040] Specifically, a group of infrared human body sensing components can be respectively set at multiple target positions in the target area. The image acquisition module can divide the target area into multiple sub-areas, and obtain the historical image information of the target area, and determine the activity frequency of the on-duty personnel in each sub-area based on the historical image information of the target area. It can be understood that in the historical image information of the target area, the on-duty personnel in different time periods can be different, and the activity frequency of the on-duty personnel in each sub-area is the activity frequency of all on-duty personnel in each sub-area. The sub-area with an activity frequency greater than a preset activity frequency threshold is taken as the target sub-area, and at least one group of infrared human body sensing components is set in each target sub-area.

[0041] Furthermore, sample data is obtained, wherein the sample data may include output signals of multiple infrared human body sensing components when the on-duty personnel are at different positions in the target area and the image acquisition unit corresponding to the position can collect the initial posture of the optimal human body image posture, and a relationship map is established based on the sample data, wherein the relationship map can characterize the relationship between the multiple initial postures and the corresponding output signals of the multiple infrared human body sensing components. Figure 7 is a schematic diagram of a relationship map according to some embodiments of this specification, such as Figure 7As shown, when infrared human body sensing component 1 and infrared human body sensing component 2 detect a human body, and other infrared human body sensing components do not detect a human body, the corresponding initial posture is initial posture 1. When infrared human body sensing component 1, infrared human body sensing component 3 and infrared human body sensing component 4 detect a human body, and other infrared human body sensing components do not detect a human body, the corresponding initial posture is initial posture 2. After acquiring the image information of the marker, the image acquisition module can directly determine the initial posture based on the output signals and relationship maps of multiple infrared human body sensing components.

[0042] It can be understood that, in the initial posture, the first image information of the target area collected by the image acquisition unit does not necessarily contain the complete facial features of the human body. Therefore, when the facial features are judged to be incomplete based on the first image information of the target area collected by the image acquisition unit, the target posture can be determined based on the first image information of the target area through a posture determination model, wherein the posture determination model can be an artificial neural network (ANN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, a bidirectional recurrent neural network (BRNN) model, and other machine learning models.

[0043] The environmental monitoring module can be used to obtain environmental information of the target area.

[0044] In some embodiments, the environmental monitoring module includes a visibility monitoring unit and a meteorological information acquisition unit, wherein the visibility monitoring unit includes a photoelectric visibility sensor, an ultrasonic visibility sensor and a laser visibility sensor, and the meteorological information acquisition unit is used to obtain meteorological information of the target area. The photoelectric visibility sensor mainly measures the intensity of atmospheric scattered light near the ground, thereby indirectly reflecting the visibility of the target area. The ultrasonic visibility sensor can measure particulate matter in the air through ultrasonic technology, thereby indirectly reflecting the atmospheric visibility of the target area as an indicator. The laser visibility sensor is a newer detection technology that detects microparticles in the air through laser irradiation, thereby achieving high-precision measurement of the visibility of the target area and gas pollution conditions such as fog, haze, and smoke.

[0045] The image processing module can be used to preprocess the image information of the target area based on the environmental information of the target area to generate preprocessed image information.

[0046] In some embodiments, the image processing module pre-processes the image information of the target area based on the environmental information of the target area to generate pre-processed image information, including:

[0047] Determine the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area;

[0048] When the visibility of the target area is less than a preset visibility threshold, the second image information of the target area is corrected by the image correction model to generate preprocessed image information.

[0049] Figure 4 is a schematic diagram of a process for determining the visibility of a target area according to some embodiments of this specification, such as Figure 4 As shown, in some embodiments, the image processing module determines the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area, including:

[0050] Determine the first visibility of the target area based on the image information of the marker and the pre-stored image information;

[0051] determining a second visibility of the target area based on an output signal of the photoelectric visibility sensor, an output signal of the ultrasonic visibility sensor, and an output signal of the laser visibility sensor;

[0052] Determine the third visibility of the target area based on the meteorological information of the target area;

[0053] The visibility of the target area is determined based on the first visibility, the second visibility, and the third visibility.

[0054] Figure 5 is a schematic diagram of a process for determining a first visibility of a target area according to some embodiments of this specification, such as Figure 5 As shown, the image processing module determines the first visibility of the target area based on the image information of the marker and the pre-stored image information, including:

[0055] Based on the image information of the marker, a first region of interest image is acquired, wherein the first region of interest image is an image of the marker in the image information of the marker;

[0056] Based on the pre-stored image information, acquiring a second region of interest image, wherein the second region of interest image is an image of a marker in the pre-stored image information;

[0057] Based on the first region of interest image, determine a first pixel RGB value matrix and a first edge pixel coordinate matrix, wherein a row vector of the first pixel RGB value matrix is ​​formed by RGB values ​​of a row of pixels of the first region of interest image arranged according to pixel positions, and a row vector in the first edge pixel coordinate matrix represents a row of pixels of the first region of interest image, wherein an element in the row vector corresponding to a non-marker edge pixel is 0, and a value of an element corresponding to a marker edge pixel is 1;

[0058] Based on the second region of interest image, determine a second pixel RGB value matrix and a second edge pixel coordinate matrix, wherein a row vector of the second pixel RGB value matrix is ​​formed by RGB values ​​of a row of pixels of the second region of interest image arranged according to pixel positions, and a row vector in the second edge pixel coordinate matrix represents a row of pixels of the second region of interest image, wherein an element in the row vector corresponding to a non-edge pixel is 0, and a value of an element corresponding to an edge pixel is 1;

[0059] A first visibility of the target area is determined based on the first pixel RGB value matrix, the first edge pixel coordinate matrix, the second pixel RGB value matrix, and the second edge pixel coordinate matrix.

[0060] Specifically, the first visibility of the target area can be determined based on the following formula:

[0061] V 1 =a 11 *S RGB +a 12 *S edge

[0062]

[0063]

[0064] Among them, V 1 is the first visibility of the target area, S RGB is the similarity between the first pixel RGB value matrix and the second pixel RGB value matrix, S edge is the similarity between the first edge pixel coordinate matrix and the second edge pixel coordinate matrix, M 1 、M 2 is the preset parameter, a 11 and a 12 For preset weights, RGB (1,ij) is the value of the element in the i-th row and j-th column of the first pixel RGB value matrix, I is the total number of rows in the first pixel RGB value matrix and the second pixel RGB value matrix, J is the total number of columns in the first pixel RGB value matrix and the second pixel RGB value matrix, RGB (2,ij) is the value of the element in the i-th row and j-th column of the second pixel RGB value matrix, E(1,pq) is the value of the element in the pth row and qth column of the first edge pixel coordinate matrix, E (2,pq) is the value of the element in the pth row and qth column in the second edge pixel coordinate matrix, P is the total number of rows of the first edge pixel coordinate matrix and the second edge pixel coordinate matrix, and Q is the total number of columns of the first edge pixel coordinate matrix and the second edge pixel coordinate matrix.

[0065] In some embodiments, the image processing module can determine the second visibility of the target area based on the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, and the output signal of the laser visibility sensor through a first visibility determination model, and determine the third visibility of the target area based on the meteorological information of the target area based on the second visibility determination model.

[0066] In some embodiments, the image processing module may determine the visibility of the target area by first normalizing the first visibility, the second visibility, and the third visibility, and performing weighted summation on the normalized first visibility, the second visibility, and the third visibility.

[0067] In some embodiments, the input of the image correction model may include the image information of the marker, the pre-stored image information, the visibility of the target area, and the second image information of the target area, and the output of the image correction model may include the corrected second image information of the target area. It is understandable that when the visibility is low, the features of the collected second image information of the target area may be missing. By correcting the image correction model based on the image information of the marker, the pre-stored image information, the visibility of the target area, and the second image information of the target area, at least part of the missing image features can be supplemented, making the subsequent off-post detection more accurate.

[0068] The first visibility determination model and the second visibility determination model may be machine learning models such as an artificial neural network (ANN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, a bidirectional recurrent neural network (BRNN) model, etc. The image correction model may be a generative adversarial network (GAN) model.

[0069] The off-duty detection module can be used to determine whether off-duty personnel have occurred in the target area based on pre-processed image information and voice information.

[0070] Figure 6is a flow chart of determining whether a duty off-duty occurs in a target area according to some embodiments of this specification, such as Figure 6 As shown, the off-duty detection module determines whether an on-duty person is off-duty in the target area based on the pre-processed image information and voice information, including:

[0071] When the image acquisition module determines that there is no human body in the target area based on the output signals of multiple infrared human body sensing components, the duration of absence from the post is recorded. When the duration of absence from the post is greater than the preset time length threshold, it is determined that the target area is absent from duty;

[0072] The off-duty detection module determines whether a person is off-duty in the target area based on the pre-processed image information and voice information, and also includes:

[0073] Based on the pre-processed image information, facial features are extracted, and based on the facial features, it is determined whether the preset on-duty personnel are on duty. If it is determined that the preset on-duty personnel are not on duty, it is determined that the target area is off duty;

[0074] When it is determined that the preset on-duty personnel are on duty, the clothing features are extracted based on the pre-processed image information to determine whether the clothing features meet the preset dress requirements. When it is determined that the clothing features do not meet the preset dress requirements, it is determined that the target area is off-duty.

[0075] In some embodiments, the off-post detection module is further used to:

[0076] When it is determined that the dress feature meets the preset dress requirement, extracting a first action feature sequence, a first expression feature sequence, and a first sound feature sequence based on preprocessed image information corresponding to a plurality of first historical time points;

[0077] Determine a first abnormal value based on the first action feature sequence, the first expression feature sequence, and the first sound feature sequence. For example, the first abnormal value may be determined based on the first action feature sequence, the first expression feature sequence, and the first sound feature sequence based on an abnormal value determination model.

[0078] When the first abnormal value is greater than the preset abnormal value threshold, a prompt message is issued;

[0079] Extracting a second action feature sequence, a second expression feature sequence, and a second sound feature sequence based on preprocessed image information corresponding to a plurality of second historical time points, wherein the second time point is after the prompt information is issued;

[0080] Based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, a second abnormal value is determined, and when the second abnormal value is greater than a preset abnormal value threshold, an alarm message is issued.

[0081] In some embodiments, the off-post detection module determines the second abnormal value based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, including:

[0082] Based on the knowledge graph, determine the matching degree between the second action feature sequence, the second expression feature sequence, and the second sound feature sequence and the prompt information;

[0083] A second abnormal value is determined based on the matching degree between the second action feature sequence, the second expression feature sequence, the second sound feature sequence and the prompt information.

[0084] Specifically, the off-duty detection module can determine the abnormality type based on the first action feature sequence, the first expression feature sequence and the first sound feature sequence, for example, the on-duty personnel are in a sleeping state, the on-duty personnel perform dangerous actions, the on-duty personnel are attacked or the on-duty personnel are physically ill, etc. Further, the corresponding prompt information is determined according to the abnormality type. For example, if the abnormality type is that the on-duty personnel are in a sleeping state, the prompt information can be a voice message "Please open your eyes and stay awake". For another example, if the abnormality type is that the on-duty personnel perform dangerous actions, the prompt information can be a voice message "Please stop the current action". For another example, if the abnormality type is that the on-duty personnel are attacked, the prompt information can be a voice message "Please reply whether it is safe".

[0085] The knowledge graph can characterize the matching degree between the prompt information and different action features, expression features and sound features. It can be understood that the matching degree between the action features, expression features and sound features of the prompt information fed back by normal on-duty personnel and the prompt information is greater than the action features, expression features and sound features of the prompt information fed back by abnormal on-duty personnel. The off-duty detection module can determine the matching degree between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information based on the knowledge graph, and determine the weights corresponding to different types of features based on the abnormality type. For example, when the abnormality type is that the on-duty personnel are in a sleeping state, the weights corresponding to the expression features and action features are greater than the weights corresponding to the sound features. Based on the weights corresponding to different types of features, the matching degrees between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information are weighted and summed to determine the second abnormal value.

[0086] Among them, the outlier determination model can be a machine learning model such as an artificial neural network (ANN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, a bidirectional recurrent neural network (BRNN) model, etc. The image correction model can be a generative adversarial network (GAN) model.

[0087] Figure 2 is a flow chart of a method for intelligently detecting off-duty personnel based on machine vision according to some embodiments of this specification, such as Figure 2 As shown, a method for intelligent detection of off-duty personnel based on machine vision may include the following process.

[0088] Step 210: Acquire image information and voice information of the target area.

[0089] Step 220, obtaining environmental information of the target area.

[0090] Step 230 , preprocessing the image information of the target area based on the environmental information of the target area to generate preprocessed image information.

[0091] Step 240, based on the pre-processed image information and voice information, determine whether there is any absence from duty in the target area.

[0092] In some embodiments, a method for detecting an on-duty off-post based on machine vision can be performed by a system for detecting an on-duty off-post based on machine vision. For more information about a method for detecting an on-duty off-post based on machine vision, see Figure 1 The related descriptions will not be repeated here.

[0093] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0094] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0095] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0096] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0097] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An intelligent detection system for off-duty personnel based on machine vision, characterized in that: include: An image acquisition module is used to acquire image information and voice information of a target area; An environmental monitoring module, used to obtain environmental information of the target area; An image processing module, used for preprocessing the image information of the target area based on the environmental information of the target area to generate preprocessed image information; An off-duty detection module, used to determine whether an off-duty person occurs in the target area based on the pre-processed image information and voice information; The image acquisition module includes a human body sensing unit, an image acquisition unit and a voice acquisition unit, wherein the human body sensing unit includes a plurality of infrared human body sensing components arranged in the target area; The image acquisition module acquires image information and voice information of the target area, including: In a target detection cycle, judging whether there is a human body in the target area based on the output signals of the plurality of infrared human body sensing components; When it is determined that there is a human body in the target area, the image acquisition unit is turned on to acquire image information of a marker in the target area at a preset position; After acquiring the image information of the marker, an initial position and posture are determined based on the output signals of the plurality of infrared human body sensing components, and the image acquisition unit is adjusted to the initial position and posture to acquire first image information of the target area; Based on the first image information of the target area, determine the target posture, adjust the image acquisition unit to the target posture, and acquire second image information of the target area; Turning on the voice collection unit to obtain voice information of the target area; The off-duty detection module determines whether off-duty occurs in the target area based on the pre-processed image information and voice information, including: When it is determined based on the preprocessed image information that the clothing feature meets the preset clothing requirement, extracting a first action feature sequence, a first expression feature sequence, and a first sound feature sequence based on the preprocessed image information corresponding to a plurality of first historical time points; Determining a first abnormal value based on the first action feature sequence, the first expression feature sequence, and the first sound feature sequence; When the first abnormal value is greater than a preset abnormal value threshold, a prompt message is issued; Extracting a second action feature sequence, a second expression feature sequence, and a second sound feature sequence based on the preprocessed image information corresponding to a plurality of second historical time points, wherein the second historical time point is after the prompt information is issued; Determine a second abnormal value based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, and issue an alarm message when the second abnormal value is greater than a preset abnormal value threshold; The off-post detection module determines a second abnormal value based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, including: Based on the knowledge graph, determining the matching degree between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information; The second abnormal value is determined based on the matching degree between the second action feature sequence, the second expression feature sequence, and the second sound feature sequence and the prompt information.

2. According to the machine vision-based intelligent detection system for off-duty personnel according to claim 1, it is characterized in that: The environmental monitoring module includes a visibility monitoring unit and a meteorological information acquisition unit, wherein the visibility monitoring unit includes a photoelectric visibility sensor, an ultrasonic visibility sensor and a laser visibility sensor, and the meteorological information acquisition unit is used to acquire the meteorological information of the target area.

3. According to the machine vision-based intelligent detection system for off-duty personnel according to claim 2, it is characterized in that: The image processing module preprocesses the image information of the target area based on the environmental information of the target area to generate preprocessed image information, including: Determining the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area; When the visibility of the target area is less than a preset visibility threshold, the second image information of the target area is corrected by an image correction model to generate the preprocessed image information.

4. According to the machine vision-based intelligent detection system for off-duty personnel according to claim 3, it is characterized in that: The image processing module determines the visibility of the target area based on the image information of the marker, the output signal of the photoelectric visibility sensor, the output signal of the ultrasonic visibility sensor, the output signal of the laser visibility sensor and the meteorological information of the target area, including: Determining a first visibility of the target area based on the image information of the marker and pre-stored image information; Determining a second visibility of the target area based on an output signal of the photoelectric visibility sensor, an output signal of the ultrasonic visibility sensor, and an output signal of the laser visibility sensor; Determining a third visibility of the target area based on the meteorological information of the target area; The visibility of the target area is determined based on the first visibility, the second visibility, and the third visibility.

5. According to the machine vision-based intelligent detection system for off-duty personnel according to claim 4, it is characterized in that: The image processing module determines the first visibility of the target area based on the image information of the marker and the pre-stored image information, including: Based on the image information of the marker, acquiring a first region of interest image; Based on the pre-stored image information, acquiring a second region of interest image; Based on the first region of interest image, determining a first pixel RGB value matrix and a first edge pixel coordinate matrix; Based on the second region of interest image, determining a second pixel RGB value matrix and a second edge pixel coordinate matrix; A first visibility of the target area is determined based on the first pixel RGB value matrix, the first edge pixel coordinate matrix, the second pixel RGB value matrix, and the second edge pixel coordinate matrix.

6. A machine vision-based intelligent detection system for off-duty personnel according to any one of claims 2 to 5, characterized in that: The off-duty detection module determines whether off-duty occurs in the target area based on the pre-processed image information and voice information, including: When the image acquisition module determines that there is no human body in the target area based on the output signals of the multiple infrared human body sensing components, the absence duration is recorded; when the absence duration is greater than a preset time length threshold, it is determined that the target area is absent from duty; The off-duty detection module determines whether the target area is off-duty based on the pre-processed image information and voice information, and further includes: Extracting facial features based on the preprocessed image information, determining whether a preset on-duty person is on duty based on the facial features, and if it is determined that the preset on-duty person is not on duty, judging that an on-duty person is off duty in the target area; When it is determined that the preset on-duty personnel are on duty, based on the preprocessed image information, clothing features are extracted to determine whether the clothing features meet the preset clothing requirements. When it is determined that the clothing features do not meet the preset clothing requirements, it is determined that the target area is off-duty.

7. An intelligent detection method for off-duty personnel based on machine vision, characterized in that: include: Acquire image information and voice information of the target area through a human body sensing unit, an image acquisition unit and a voice acquisition unit, wherein the human body sensing unit includes a plurality of infrared human body sensing components arranged in the target area; Acquiring environmental information of the target area; Preprocessing the image information of the target area based on the environmental information of the target area to generate preprocessed image information; Based on the pre-processed image information and voice information, determining whether a person on duty is absent from the target area; The image information and voice information of the target area are obtained, including: In a target detection cycle, judging whether there is a human body in the target area based on the output signals of the plurality of infrared human body sensing components; When it is determined that there is a human body in the target area, the image acquisition unit is turned on to acquire image information of a marker in the target area at a preset position; After acquiring the image information of the marker, an initial position and posture are determined based on the output signals of the plurality of infrared human body sensing components, and the image acquisition unit is adjusted to the initial position and posture to acquire first image information of the target area; Based on the first image information of the target area, determine the target posture, adjust the image acquisition unit to the target posture, and acquire second image information of the target area; Turning on the voice collection unit to obtain voice information of the target area; Based on the pre-processed image information and voice information, judging whether the target area is off-duty, includes: When it is determined based on the preprocessed image information that the clothing feature meets the preset clothing requirement, extracting a first action feature sequence, a first expression feature sequence, and a first sound feature sequence based on the preprocessed image information corresponding to a plurality of first historical time points; Determining a first abnormal value based on the first action feature sequence, the first expression feature sequence, and the first sound feature sequence; When the first abnormal value is greater than a preset abnormal value threshold, a prompt message is issued; Extracting a second action feature sequence, a second expression feature sequence, and a second sound feature sequence based on the preprocessed image information corresponding to a plurality of second historical time points, wherein the second historical time point is after the prompt information is issued; Determine a second abnormal value based on the second action feature sequence, the second expression feature sequence and the second sound feature sequence, and issue an alarm message when the second abnormal value is greater than a preset abnormal value threshold; Determining a second abnormal value based on the second action feature sequence, the second expression feature sequence, and the second sound feature sequence includes: Based on the knowledge graph, determining the matching degree between the second action feature sequence, the second expression feature sequence and the second sound feature sequence and the prompt information; The second abnormal value is determined based on the matching degree between the second action feature sequence, the second expression feature sequence, and the second sound feature sequence and the prompt information.