Object detection method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of image processing, is applied to a bank human body detection scene, and relates to an object detection method, device and equipment and a storage medium thereof.The object detection method comprises the steps that two detection models are adopted, integral detection and local detection are conducted on human body parts in the same detection image, and then consistency verification is conducted; the method comprises the following steps: verifying a human body monitoring video clip of a bank, determining the risk of missing detection / false detection according to a verification result, selecting different subsequent processing measures according to the risk of missing detection / false detection, applying the object detection method to a human body recognition scene of the bank, and particularly detecting a human body target in the human body monitoring video clip of the bank by adopting the object detection method. By means of the method, missing detection / false detection targets can be rapidly screened out from the monitoring video, related processing measures are given, a bank security department is assisted in bank security protection, and bank security is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and is applied to human body detection scenarios in banks, and in particular to an object detection method, device, equipment and storage medium thereof. Background Art

[0002] In the personnel identification and detection tasks of banks, two situations may occur: missed detection and false detection. Missed detection refers to the system's failure to correctly identify and detect real people. This may happen when the characteristics of the person are not obvious or the environmental conditions are complex. For example, when the customer population is dense and the flow is large, missed detection may occur. In this case, the system may not be able to provide accurate security monitoring, which will affect the security of the bank. False detection means that the system mistakenly identifies non-target people as target people. This may happen when the recognition algorithm identifies human-like objects as human bodies, such as human models.

[0003] At present, in order to cope with these situations, banks need to continuously improve and optimize recognition algorithms in personnel identification and detection tasks, and improve their adaptability to complex scenarios and special situations. However, blindly optimizing and improving the detection model not only requires a lot of training data and energy, resulting in high investment costs, but also the final inspection results cannot be effectively guaranteed, and comparative verification cannot be formed, resulting in the inability to quickly screen out missed / falsely detected targets, and the auxiliary effect on the bank's security department is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an object detection method, device, equipment and storage medium thereof to solve the problem that the prior art cannot quickly screen out missed / falsely detected targets in bank human target detection scenarios, and has a low auxiliary effect on bank security departments.

[0005] In order to solve the above technical problems, the present application provides an object detection method, which adopts the following technical solutions:

[0006] An object detection method comprises the following steps:

[0007] Get the video clip to be detected;

[0008] Extracting a target number of images to be detected from the video clip to be detected;

[0009] Inputting the target number of images to be detected into the first detection model and the second detection model respectively in sequence;

[0010] Obtaining the overall detection result output by the first detection model; and,

[0011] Obtaining a locality detection result output by the second detection model;

[0012] The overall detection result and the local detection result corresponding to the same image to be detected are input into a preset consistency verification model;

[0013] Predict the missed detection / false detection risk of each image to be detected based on the preset verification strategy and risk assessment strategy;

[0014] Target processing measures are screened out based on the risk of missed detection / false detection for each image to be detected.

[0015] Furthermore, the first detection model includes a human joint detection model based on OpenPose, and the step of obtaining the overall detection result output by the first detection model specifically includes:

[0016] Using the feature extraction component of the human joint point detection model to extract features from the current detection image to obtain a feature extraction result;

[0017] Inputting the feature extraction result into a feature key point screening component to screen out feature key points, and marking the feature key points with human body part information according to prior knowledge;

[0018] Input the feature key points into the key point connection component to obtain a feature key point linear connection graph;

[0019] The linear connection diagram of the feature key points is used as the overall detection result corresponding to the current image to be detected.

[0020] Furthermore, the feature extraction component of the human joint point detection model is composed of a CNN convolutional neural network, and the step of using the feature extraction component of the human joint point detection model to extract features from the current detection image to obtain a feature extraction result specifically includes:

[0021] Performing convolution processing on the current detection image through the CNN convolutional neural network to obtain the feature extraction result;

[0022] The step of inputting the feature extraction result into the feature key point screening component, screening out the feature key points, and marking the feature key points with human body part information according to prior knowledge specifically includes:

[0023] Inputting the feature extraction result into the pooling layer connected to the CNN convolutional neural network for downsampling processing;

[0024] Obtain the image heat map corresponding to the target channel from the downsampling processing result;

[0025] Selecting feature points whose thermal values ​​exceed a preset thermal value threshold from the image thermal map as preliminary feature key points, wherein the thermal value refers to a pixel value;

[0026] A preset position compensation mechanism is used to perform position compensation on all the initially selected key feature points, and the final key feature points are determined based on the position compensation results. The human body part information corresponding to all key feature points is marked based on prior knowledge.

[0027] The step of inputting the feature key points into the key point connection component to obtain a linear connection graph of the feature key points specifically includes:

[0028] According to the RAF position and limb association method in OpenPose, the final feature key points are connected to obtain the linear connection diagram of the feature key points.

[0029] Furthermore, after executing the step of obtaining the integrity test result output by the first test model, the method further includes:

[0030] Analyze and count the linear connection graph of the feature key points corresponding to the current detection image;

[0031] Counting the number of human body line drawings included in the characteristic key point linear connection diagram, and numbering all human body line drawings to obtain first numbering information; and,

[0032] Counting the number of characteristic key points contained in each human body line drawing, and numbering the characteristic key points in each human body line drawing respectively to obtain second numbering information;

[0033] Determine the first serial number information corresponding to each key feature point according to the human body line drawing where each key feature point is located;

[0034] The first numbering information and the second numbering information are concatenated as numbering identification information of the feature key point in the feature key point linear connection graph.

[0035] Furthermore, the second detection model includes a human body local area detection model based on an improved Mask-RCNN, and the step of obtaining the local detection result output by the second detection model specifically includes:

[0036] Step 501, selecting different feature key points in sequence from the feature key point linear connection graph corresponding to the current detection image as target key points;

[0037] Step 502, extracting a target human body local area image from the current detection image according to the position information of the target key point and the preset detection area size;

[0038] Step 503, using the feature map extraction component of the human body local area detection model to perform multi-scale feature map extraction and fusion on the target human body local area image to obtain a feature extraction result;

[0039] Step 504, inputting the feature extraction result into the classification prediction component to perform human body part category prediction, and obtaining a human body part category prediction result;

[0040] Step 505, repeatedly executing steps 501 to 504, obtaining the human body part category prediction results corresponding to all the feature key points in the current detection image as the local detection results;

[0041] After executing the step of obtaining the locality detection result output by the second detection model, the method further includes:

[0042] The number identification information of the target key point in the linear connection diagram of the feature key points is set as the distinguishing number information of the human body part category prediction result corresponding to the target key point.

[0043] Furthermore, the feature map extraction component of the human body local area detection model includes a feature map extraction component based on the ResNet-FPN network structure, and the step of using the feature map extraction component of the human body local area detection model to perform multi-scale feature map extraction and fusion on the target human body local area image to obtain a feature extraction result specifically includes:

[0044] Using the ResNet-FPN network structure to extract a multi-scale feature map from the local area image of the target human body to obtain a multi-scale feature map;

[0045] The multi-scale feature maps are uniformly scaled to the scale of the target human body local area image, and feature map fusion is performed to obtain the fused feature map as a feature extraction result;

[0046] The classification prediction component is composed of a mask branch network of the improved Mask-RCNN human body local area detection model, and the step of inputting the feature extraction result into the classification prediction component to perform human body part category prediction and obtain the human body part category prediction result specifically includes:

[0047] The feature extraction result is input into the mask branch network to obtain the human body part information output by the mask branch network, wherein before using the human body local area detection model, the classification prediction category of the mask branch network is pre-set to different human body parts.

[0048] Furthermore, the step of predicting the risk of missed detection / false detection of each image to be detected according to the preset verification strategy and risk assessment strategy specifically includes:

[0049] According to the overall detection result corresponding to the current image to be detected, all the characteristic key points contained in the current image to be detected are identified;

[0050] Obtaining the human body part marking information corresponding to all the key feature points, and obtaining the number identification information corresponding to all the key feature points;

[0051] According to the local detection results corresponding to the current image to be detected, the human body part category prediction results corresponding to all the feature key points in the current detection image are identified, and the distinguishing number information corresponding to all the human body part category prediction results is obtained;

[0052] When the distinguishing number information is the same as the number identification information, the percentage of the body part category prediction results that are consistent with the body part marking information is counted;

[0053] If the number ratio exceeds a preset first ratio threshold, there is no missed detection / false detection in the current detection image;

[0054] If the number ratio does not exceed a preset first ratio threshold but exceeds a preset second ratio threshold, then the current detection image has a low risk of missed detection / false detection, wherein the first ratio threshold is greater than the second ratio threshold;

[0055] If the number ratio does not exceed the preset second ratio threshold, the current detection image has a high risk of missed detection / false detection;

[0056] The step of selecting target processing measures according to the missed detection / false detection risk of each image to be detected specifically includes:

[0057] If there is no missed detection or false detection in the current detection image, a prompt indicating that the detection is normal is sent to the detection feedback interface;

[0058] If the current detection image has a low risk of missed detection / false detection, an instruction to continue sampling detection is sent to the detection execution end, and a prompt that the detection is continuing is sent to the detection feedback interface;

[0059] If the current detection image has a high risk of missed detection / false detection, a prompt for manual verification is sent to the detection feedback interface.

[0060] In order to solve the above technical problems, the embodiment of the present application also provides an object detection device, which adopts the following technical solution:

[0061] An object detection device, comprising:

[0062] A detection video acquisition module is used to acquire the video clip to be detected;

[0063] A module for extracting images to be detected, used to extract a target number of images to be detected from the video clip to be detected;

[0064] A detection image input module, used to input the target number of images to be detected into the first detection model and the second detection model respectively in sequence;

[0065] an integrity detection result acquisition module, used to acquire the integrity detection result output by the first detection model; and

[0066] A locality detection result acquisition module, used to obtain the locality detection result output by the second detection model;

[0067] A verification input module is used to input the overall detection result and the local detection result corresponding to the same image to be detected into a preset consistency verification model;

[0068] The detection risk prediction module is used to predict the missed detection / false detection risk of each image to be detected based on the preset verification strategy and risk assessment strategy;

[0069] The processing measure screening module is used to screen out target processing measures according to the missed detection / false detection risk of each image to be detected.

[0070] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0071] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the object detection method described above when executing the computer-readable instructions.

[0072] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0073] A computer-readable storage medium stores computer-readable instructions, which implement the steps of the object detection method described above when executed by a processor.

[0074] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0075] The object detection method described in the embodiment of the present application adopts two detection models to perform overall detection and local detection on the human body parts in the same detection image respectively, and then performs consistency verification, determines the risk of missed detection / false detection according to the verification result, and selects different subsequent processing measures according to the risk of missed detection / false detection. The object detection method is applied to the bank human body recognition scenario, especially using the object detection method to detect human targets in the bank's human body monitoring video clips, which can quickly screen out missed detection / false detection targets from the monitoring video, and provide relevant processing measures to assist the bank security department in protecting the bank's security and improve bank security. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0078] Figure 2 is a flow chart of an embodiment of an object detection method according to the present application;

[0079] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0080] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;

[0081] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0082] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 207 is shown;

[0083] Figure 7 is a schematic structural diagram of an embodiment of an object detection device according to the present application;

[0084] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0086] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0087] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0088] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0089] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0090] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0091] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0092] It should be noted that the object detection method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the object detection device is generally arranged in the terminal device.

[0093] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0094] Continue to refer Figure 2 , shows a flow chart of an embodiment of an object detection method according to the present application. The object detection method comprises the following steps:

[0095] Step 201: Obtain a video clip to be detected.

[0096] In this embodiment, the video clips to be detected specifically include: the human surveillance video clips obtained by segmenting the human surveillance video taken by the bank's external / ATM surveillance camera, the surveillance camera deployed in the bank's business hall, or the bank's identity verification camera. The main purpose is to use the object detection method to detect human targets in the bank's human surveillance video clips, quickly screen out missed / falsely detected targets from the surveillance video, and give relevant processing measures to assist the bank's security department in protecting the bank's security and improve bank security.

[0097] Step 202: extracting a target number of images to be detected from the video segment to be detected.

[0098] In the present embodiment, the target number of images to be detected are extracted from the video segment to be detected. The specific implementation method is as follows: first, the video segment to be detected is frame-processed according to a preset window and sampling frequency, and then, key frames are screened out from the frame processing results, and the images corresponding to all the key frames are sorted out as the target number of images to be detected.

[0099] By dividing the frames and extracting key frames, we can obtain the target number of images to be detected, ensuring the detection of important surveillance images in the bank's surveillance video, preventing the bank from having security omissions, and ensuring that business interactions and processing within the bank are under safe monitoring.

[0100] Step 203: input the target number of images to be detected into the first detection model and the second detection model respectively in sequence.

[0101] In the present embodiment, the same image to be detected is detected using the first detection model and the second detection model respectively, thereby avoiding the problem of not being able to timely discover missed detections / false detections when a single detection model is used for detection. Moreover, the use of two detection models for detection can also verify each other based on the detection results, thereby quickly identifying missed detections / false detections, so that timely remediation and safety measures can be taken.

[0102] Step 204: Obtain the integrity detection result output by the first detection model.

[0103] In this embodiment, the first detection model includes a human joint point detection model based on OpenPose, wherein OpenPose is an open source real-time multi-person 2D posture estimation library, which uses deep learning technology, can detect key points of the human body, face and hand from RGB images or videos, supports the detection of up to 135 key points, can detect the postures of multiple people in a single image at the same time, and can identify and locate key points of the human body, face and hand, including elbows, knees, shoulders, hips, eyes, noses, mouths and fingertips, etc. In this embodiment, the human joint point detection model based on OpenPose is used. First, each feature key point in the target image (image to be detected) is detected, and then the feature key points are connected to obtain a linear connection diagram of the feature key points as the overall detection result corresponding to the current image to be detected. By using the linear connection diagram of the feature key points as the overall detection result corresponding to the current image to be detected, it is possible to detect multiple people in the monitoring screen at the same time, and to more intuitively represent the body posture of each human body monomer in a feature key point graphic manner, so that the detection is more comprehensive.

[0104] Continue to refer Figure 3 , Figure 3 yes Figure 2The flowchart of a specific embodiment of step 204 shown includes:

[0105] Step 301, using the feature extraction component of the human joint point detection model to perform feature extraction on the current detection image to obtain a feature extraction result;

[0106] In this embodiment, the feature extraction component of the human joint point detection model is composed of a CNN convolutional neural network, and the step of using the feature extraction component of the human joint point detection model to extract features of the current detection image to obtain the feature extraction result specifically includes: performing convolution processing on the current detection image through the CNN convolutional neural network to obtain the feature extraction result.

[0107] Step 302, inputting the feature extraction result into a feature key point screening component, screening out feature key points, and marking the feature key points with human body part information according to prior knowledge;

[0108] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes:

[0109] Step 401, inputting the feature extraction result into a pooling layer connected to the CNN convolutional neural network for downsampling processing;

[0110] Step 402, obtaining an image heat map corresponding to the target channel from the downsampling processing result;

[0111] Step 403, selecting feature points whose thermal values ​​exceed a preset thermal value threshold from the image thermal map as primary feature key points, wherein the thermal value refers to a pixel value;

[0112] Step 404, using a preset position compensation mechanism to perform position compensation on all initially selected feature key points, and determining the final feature key points according to the position compensation results;

[0113] Step 405: Mark the human body part information corresponding to all feature key points according to prior knowledge.

[0114] Specifically, the target channel refers to any image extraction channel after downsampling, such as the R, G, and B channels in the RGB three channels; the feature points whose thermal values ​​exceed the preset thermal value threshold are selected from the image thermal map as the pre-selected feature key points, that is, the feature points whose pixel values ​​exceed the preset pixel value threshold are selected from the image thermal map as the pre-selected feature key points; the preset position compensation mechanism varies according to the angle and position of the camera. Since the detection image is extracted from the surveillance video clip shot by the camera, there will always be a spatial error between the position at the time of shooting and the real position or coordinates. The position compensation mechanism is used to compensate the position of the feature key points, the purpose of which is to ensure that the final feature key point position is more accurate. The feature key points are marked with human body part information based on prior knowledge, which is convenient for subsequent detection and verification with the detection results of the second detection model. The prior knowledge refers to the head information, left hand part, right hand part, elbow joint part, shoulder joint part and other human body part information in the known human posture structure diagram, which will not be repeated here.

[0115] Step 303, inputting the feature key points into the key point connection component to obtain a feature key point linear connection graph;

[0116] In this embodiment, the step of inputting the feature key points into the key point connection component to obtain the feature key point linear connection diagram specifically includes: according to the RAF position and limb association method in OpenPose, connecting the final feature key points to obtain the feature key point linear connection diagram.

[0117] Specifically, OpenPose includes two detection methods, one is a top-down detection strategy, and the other is a bottom-up detection strategy. In this embodiment, a bottom-up detection strategy, namely, the PAF (Part Affinity Fields) detection strategy, is adopted. This detection strategy encodes the position and limb direction information through a set of 2D vector fields, so that the feature key points can be connected according to the position and limb direction information, that is, according to the RAF position and limb association method in OpenPose, the final feature key points are connected to obtain the linear connection diagram of the feature key points. By adopting the RAF position and limb association method in OpenPose to connect the final feature key points, more accurate human posture estimation is achieved.

[0118] Step 304: Use the linear connection diagram of the feature key points as the overall detection result corresponding to the current image to be detected.

[0119] In this embodiment, after executing the step of obtaining the overall detection result output by the first detection model, the method also includes: analyzing and counting the linear connection diagram of the feature key points corresponding to the current detection image; counting the number of human body line diagrams contained in the linear connection diagram of the feature key points, and numbering all the human body line diagrams to obtain first numbering information; and counting the number of feature key points contained in each human body line diagram, and numbering the feature key points in each human body line diagram respectively to obtain second numbering information; determining the first numbering information corresponding to each feature key point according to the human body line diagram where each feature key point is located; and splicing the first numbering information and the second numbering information as the numbering identification information of the feature key point in the linear connection diagram of the feature key point.

[0120] Specifically, the characteristic key points in the current detection image are numbered from two dimensions of different human bodies and different key points, so as to facilitate the subsequent identification of key points corresponding to different human bodies in the image.

[0121] Step 205: Obtain the locality detection result output by the second detection model.

[0122] In this embodiment, the second detection model includes a human body local area detection model based on the improved Mask-RCNN. Specifically, the improved Mask-RCNN refers to removing the RPN candidate frame extraction network in the conventional Mask-RCNN neural network, and setting the specified detection frame extraction component before the processing step of the ResNet-FPN network structure in the conventional Mask-RCNN neural network, that is, the detection frame extraction component is pre-placed before the ResNet-FPN network structure in the conventional Mask-RCNN neural network, and the RPN candidate frame extraction network is deleted. The purpose is to reduce the candidate frame selection training and prediction of the Mask-RCNN neural network, and directly use the detection frame extraction component to extract a local detection image with accurate position and fixed size from the target image to detect the local position information of the human body.

[0123] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:

[0124] Step 501, selecting different feature key points in sequence from the feature key point linear connection graph corresponding to the current detection image as target key points;

[0125] Step 502, extracting a target human body local area image from the current detection image according to the position information of the target key point and the preset detection area size;

[0126] Specifically, the extraction of the target human body local area image from the current detection image based on the position information of the target key point and the preset detection area size is completed by the front detection frame extraction component, wherein the setting parameters of the front detection frame extraction component include the position information of the target key point and the preset detection area size, and the detection area can be a frame-shaped area with the position information of the target key point as the center point, or a circular area with the position information of the target key point as the center point, which is specifically set according to the actual detection situation.

[0127] Step 503, using the feature map extraction component of the human body local area detection model to perform multi-scale feature map extraction and fusion on the target human body local area image to obtain a feature extraction result;

[0128] In this embodiment, the feature map extraction component of the human body local area detection model includes a feature map extraction component based on the ResNet-FPN network structure, wherein "ResNet" refers to a residual network structure, "FPN" refers to a pyramid-like multi-level feature extraction algorithm for images, and the ResNet-FPN network structure uses a residual network to use the FPN image feature extraction algorithm to extract features from the target human body local area image, and the extracted multi-scale feature map is similar to a special pyramid hierarchical graph structure. The feature map extraction component based on the ResNet-FPN network structure is used to extract multi-scale feature maps from the target human body local area image, thereby realizing image feature extraction from multiple scales, making the feature extraction result more accurate.

[0129] Specifically, the step of using the feature map extraction component of the human body local area detection model to perform multi-scale feature map extraction and fusion on the target human body local area image to obtain a feature extraction result includes: using the ResNet-FPN network structure to perform multi-scale feature map extraction on the target human body local area image to obtain a multi-scale feature map; uniformly scaling the multi-scale feature map to the scale of the target human body local area image, and performing feature map fusion to obtain the fused feature map as the feature extraction result.

[0130] By uniformly scaling the multi-scale feature map to the scale of the target human body local area image and performing feature map fusion, a fused feature map is obtained as a feature extraction result, so that the finally obtained fused feature map has richer feature information.

[0131] Step 504, inputting the feature extraction result into the classification prediction component to perform human body part category prediction, and obtaining a human body part category prediction result;

[0132] In this embodiment, the classification prediction component is composed of a mask branch network of the improved Mask-RCNN human body local area detection model. Specifically, in this embodiment, the mask branch network plays a role in encoding and predicting the feature extraction results, and predicting the human body part information corresponding to the human body local area image. It can be regarded as a classification network.

[0133] Specifically, the step of inputting the feature extraction result into the classification prediction component to perform human body part category prediction and obtaining the human body part category prediction result includes: inputting the feature extraction result into the mask branch network, obtaining the human body part information output by the mask branch network, wherein, before using the human body local area detection model, the classification prediction category of the mask branch network is pre-set to different human body parts.

[0134] Step 505, repeatedly executing steps 501 to 504, and obtaining the human body part category prediction results corresponding to all the feature key points in the current detection image as the local detection results.

[0135] In this embodiment, after executing the step of obtaining the local detection results output by the second detection model, the method also includes: setting the number identification information of the target key point in the linear connection graph of the feature key points to the distinctive number information of the human body part category prediction result corresponding to the target key point.

[0136] Step 206: Input the overall detection result and the local detection result corresponding to the same image to be detected into a preset consistency verification model.

[0137] Step 207 : predicting the missed detection / false detection risk of each image to be detected according to the preset verification strategy and risk assessment strategy.

[0138] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 207 shown includes:

[0139] Step 601, identifying all feature key points contained in the current image to be detected according to the overall detection result corresponding to the current image to be detected;

[0140] Step 602, obtaining the human body part labeling information corresponding to all the key feature points, and obtaining the number identification information corresponding to all the key feature points;

[0141] Step 603, identifying the human body part category prediction results corresponding to all feature key points in the current detection image according to the local detection results corresponding to the current image to be detected, and obtaining the distinguishing number information corresponding to all the human body part category prediction results;

[0142] Step 604, counting the proportion of the body part category prediction results that are consistent with the body part label information when the distinguishing number information is the same as the number identification information;

[0143] Step 605: if the number ratio exceeds a preset first ratio threshold, there is no missed detection / false detection in the current detection image;

[0144] Specifically, under ideal conditions, the preset first ratio threshold is 100%, that is, the human body part category prediction result is completely consistent with the human body part marking information. However, the preset first ratio threshold often has a certain fault tolerance when setting, for example: the first ratio threshold is 98%.

[0145] Step 606: If the number ratio does not exceed the preset first ratio threshold but exceeds the preset second ratio threshold, the current detection image has a low risk of missed detection / false detection, wherein the first ratio threshold is greater than the second ratio threshold;

[0146] Specifically, for example: the first ratio threshold is 98%, and the second ratio threshold is 80%. At this time, there is a lower risk of missed detection / false detection. Of course, the higher the second ratio threshold value, the more accurate the object detection method is, and the lower the second ratio threshold value, the less accurate the object detection method is.

[0147] Step 607: If the number ratio does not exceed the preset second ratio threshold, the current detection image has a high risk of missed detection / false detection.

[0148] Step 208 , select target processing measures according to the missed detection / false detection risk of each image to be detected.

[0149] In this embodiment, the step of screening out target processing measures according to the risk of missed detection / false detection of each image to be detected specifically includes: if there is no missed detection / false detection in the current detected image, a prompt indicating that the detection is normal is sent to the detection feedback interface; if there is a low risk of missed detection / false detection in the current detected image, an instruction to continue sampling and detection is sent to the detection execution end, and a prompt indicating that the detection is continuing is sent to the detection feedback interface; if there is a high risk of missed detection / false detection in the current detected image, a prompt for manual verification is sent to the detection feedback interface.

[0150] By adopting two detection models, the human body parts in the same detection image are respectively subjected to overall detection and local detection, and then consistency verification is performed. The risk of missed detection / false detection is determined according to the verification result, and different subsequent processing measures are selected according to the risk of missed detection / false detection. The object detection method is applied to the bank human body recognition scenario, especially the object detection method is used to detect human targets in the bank's human body monitoring video clips, which can quickly screen out missed detection / false detection targets from the monitoring video, and give relevant processing measures to assist the bank security department in bank security protection and improve bank security.

[0151] The present application adopts two detection models to perform overall detection and local detection on human body parts in the same detection image, and then performs consistency verification. The risk of missed detection / false detection is determined according to the verification result, and different subsequent processing measures are selected according to the risk of missed detection / false detection. The object detection method is applied to bank human body recognition scenarios, especially using the object detection method to detect human targets in bank human body monitoring video clips. It can quickly screen out missed detection / false detection targets from the monitoring video, and provide relevant processing measures to assist the bank security department in protecting the bank's security and improve bank security.

[0152] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0153] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0154] In an embodiment of the present application, two detection models are used to perform overall detection and local detection on human body parts in the same detection image, and then consistency verification is performed. The risk of missed detection / false detection is determined based on the verification result, and different subsequent processing measures are selected based on the risk of missed detection / false detection. The object detection method is applied to bank human recognition scenarios, especially using the object detection method to detect human targets in bank human surveillance video clips. It is possible to quickly screen out missed detection / false detection targets from surveillance videos, and provide relevant processing measures to assist the bank security department in protecting bank security and improving bank security.

[0155] Further references Figure 7 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of an object detection device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] like Figure 7 As shown, the object detection device 700 described in this embodiment includes: a detection video acquisition module 701, a to-be-detected image extraction module 702, a detection image input module 703, an overall detection result acquisition module 704, a local detection result acquisition module 705, a verification input module 706, a detection risk prediction module 707 and a treatment measure screening module 708. Among them:

[0157] The detection video acquisition module 701 is used to acquire the video segment to be detected;

[0158] The to-be-detected image extraction module 702 is used to extract a target number of to-be-detected images from the to-be-detected video segment;

[0159] A detection image input module 703, used to input the target number of images to be detected into the first detection model and the second detection model respectively in sequence;

[0160] An integrity detection result acquisition module 704 is used to acquire the integrity detection result output by the first detection model; and

[0161] A locality detection result acquisition module 705, used to acquire the locality detection result output by the second detection model;

[0162] The verification input module 706 is used to input the overall detection result and the local detection result corresponding to the same image to be detected into a preset consistency verification model;

[0163] The detection risk prediction module 707 is used to predict the missed detection / false detection risk of each image to be detected according to the preset verification strategy and risk assessment strategy;

[0164] The processing measure screening module 708 is used to screen out target processing measures according to the missed detection / false detection risk of each image to be detected.

[0165] The object detection device proposed in the present application belongs to the field of image processing technology and is applied to human body detection scenarios in banks. By adopting two detection models, overall detection and local detection are respectively performed on human body parts in the same detection image, and then consistency verification is performed. The risk of missed detection / false detection is determined according to the verification result, and different subsequent processing measures are selected according to the risk of missed detection / false detection. The object detection method is applied to human body recognition scenarios in banks, especially the object detection method is used to detect human targets in bank human body monitoring video clips, which can quickly screen out missed detection / false detection targets from the monitoring video and give relevant processing measures to assist the bank security department in bank security protection and improve bank security.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0168] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0169] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 having components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0170] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0171] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of an object detection method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0172] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run the computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the object detection method.

[0173] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0174] The computer device proposed in this embodiment belongs to the field of image processing technology and is applied to the human body detection scenario in a bank. This application adopts two detection models to perform overall detection and local detection on the human body parts in the same detection image, and then performs consistency verification, determines the risk of missed detection / false detection according to the verification results, and selects different subsequent processing measures according to the risk of missed detection / false detection. The object detection method is applied to the human body recognition scenario in the bank, especially the object detection method is used to detect human targets in the human body monitoring video clips of the bank, which can quickly screen out missed detection / false detection targets from the monitoring video, and give relevant processing measures to assist the bank security department in bank security protection and improve bank security.

[0175] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the object detection method as described above.

[0176] The computer-readable storage medium proposed in this embodiment belongs to the field of image processing technology and is applied to the human body detection scenario in a bank. This application adopts two detection models to perform overall detection and local detection on the human body parts in the same detection image, and then performs consistency verification, determines the risk of missed detection / false detection according to the verification results, and selects different subsequent processing measures according to the risk of missed detection / false detection. The object detection method is applied to the human body recognition scenario in the bank, especially the object detection method is used to detect human targets in the human body monitoring video clips of the bank, which can quickly screen out missed detection / false detection targets from the monitoring video, and give relevant processing measures to assist the bank security department in bank security protection and improve bank security.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An object detection method, characterized in that: The steps include: Get the video clip to be detected; Extracting a target number of images to be detected from the video clip to be detected; Inputting the target number of images to be detected into the first detection model and the second detection model respectively in sequence; Obtaining an overall detection result output by the first detection model; as well as, Obtaining a locality detection result output by the second detection model; The overall detection result and the local detection result corresponding to the same image to be detected are input into a preset consistency verification model; Predict the missed detection / false detection risk of each image to be detected based on the preset verification strategy and risk assessment strategy; Target processing measures are screened out based on the risk of missed detection / false detection for each image to be detected.

2. The object detection method according to claim 1, characterized in that: The first detection model includes a human joint point detection model based on OpenPose, and the step of obtaining the overall detection result output by the first detection model specifically includes: Using the feature extraction component of the human joint point detection model to extract features from the current detection image to obtain a feature extraction result; Inputting the feature extraction result into a feature key point screening component to screen out feature key points, and marking the feature key points with human body part information according to prior knowledge; Input the feature key points into the key point connection component to obtain a feature key point linear connection graph; The linear connection diagram of the feature key points is used as the overall detection result corresponding to the current image to be detected.

3. The object detection method according to claim 2, characterized in that: The feature extraction component of the human joint point detection model is composed of a CNN convolutional neural network, and the step of using the feature extraction component of the human joint point detection model to extract features from the current detection image to obtain a feature extraction result specifically includes: Performing convolution processing on the current detection image through the CNN convolutional neural network to obtain the feature extraction result; The step of inputting the feature extraction result into the feature key point screening component, screening out the feature key points, and marking the feature key points with human body part information according to prior knowledge specifically includes: Inputting the feature extraction result into the pooling layer connected to the CNN convolutional neural network for downsampling processing; Obtain the image heat map corresponding to the target channel from the downsampling processing result; Selecting feature points whose thermal values ​​exceed a preset thermal value threshold from the image thermal map as preliminary feature key points, wherein the thermal value refers to a pixel value; A preset position compensation mechanism is used to perform position compensation on all the initially selected key feature points, and the final key feature points are determined based on the position compensation results. The human body part information corresponding to all key feature points is marked based on prior knowledge. The step of inputting the feature key points into the key point connection component to obtain a linear connection graph of the feature key points specifically includes: According to the RAF position and limb association method in OpenPose, the final feature key points are connected to obtain the linear connection diagram of the feature key points.

4. The object detection method according to claim 2, characterized in that: After executing the step of obtaining the integrity test result output by the first test model, the method further includes: Analyze and count the linear connection graph of the feature key points corresponding to the current detection image; Counting the number of human body line drawings included in the characteristic key point linear connection diagram, and numbering all human body line drawings to obtain first numbering information; and, Counting the number of characteristic key points contained in each human body line drawing, and numbering the characteristic key points in each human body line drawing respectively to obtain second numbering information; Determine the first serial number information corresponding to each key feature point according to the human body line drawing where each key feature point is located; The first numbering information and the second numbering information are concatenated as numbering identification information of the feature key point in the feature key point linear connection graph.

5. The object detection method according to claim 4, characterized in that: The second detection model includes a human body local area detection model based on an improved Mask-RCNN, and the step of obtaining the local detection result output by the second detection model specifically includes: Step 501, selecting different feature key points in sequence from the feature key point linear connection graph corresponding to the current detection image as target key points; Step 502, extracting a target human body local area image from the current detection image according to the position information of the target key point and the preset detection area size; Step 503, using the feature map extraction component of the human body local area detection model to perform multi-scale feature map extraction and fusion on the target human body local area image to obtain a feature extraction result; Step 504, inputting the feature extraction result into the classification prediction component to perform human body part category prediction, and obtaining a human body part category prediction result; Step 505, repeatedly executing steps 501 to 504, obtaining the human body part category prediction results corresponding to all the feature key points in the current detection image as the local detection results; After executing the step of obtaining the locality detection result output by the second detection model, the method further includes: The number identification information of the target key point in the linear connection diagram of the feature key points is set as the distinguishing number information of the human body part category prediction result corresponding to the target key point.

6. The object detection method according to claim 5, characterized in that: The feature map extraction component of the human body local area detection model includes a feature map extraction component based on a ResNet-FPN network structure, and the step of using the feature map extraction component of the human body local area detection model to extract and fuse multi-scale feature maps of the target human body local area image to obtain a feature extraction result specifically includes: Using the ResNet-FPN network structure to extract a multi-scale feature map from the local area image of the target human body to obtain a multi-scale feature map; The multi-scale feature maps are uniformly scaled to the scale of the target human body local area image, and feature map fusion is performed to obtain the fused feature map as a feature extraction result; The classification prediction component is composed of a mask branch network of the improved Mask-RCNN human body local area detection model, and the step of inputting the feature extraction result into the classification prediction component to perform human body part category prediction and obtain the human body part category prediction result specifically includes: The feature extraction result is input into the mask branch network to obtain the human body part information output by the mask branch network, wherein before using the human body local area detection model, the classification prediction category of the mask branch network is pre-set to different human body parts.

7. The object detection method according to claim 5, characterized in that: The step of predicting the risk of missed detection / false detection of each image to be detected according to the preset verification strategy and risk assessment strategy specifically includes: According to the overall detection result corresponding to the current image to be detected, all the characteristic key points contained in the current image to be detected are identified; Obtaining the human body part marking information corresponding to all the key feature points, and obtaining the number identification information corresponding to all the key feature points; According to the local detection results corresponding to the current image to be detected, the human body part category prediction results corresponding to all the feature key points in the current detection image are identified, and the distinguishing number information corresponding to all the human body part category prediction results is obtained; When the distinguishing number information is the same as the number identification information, the percentage of the body part category prediction results that are consistent with the body part marking information is counted; If the number ratio exceeds a preset first ratio threshold, there is no missed detection / false detection in the current detection image; If the number ratio does not exceed a preset first ratio threshold but exceeds a preset second ratio threshold, then the current detection image has a low risk of missed detection / false detection, wherein the first ratio threshold is greater than the second ratio threshold; If the number ratio does not exceed the preset second ratio threshold, the current detection image has a high risk of missed detection / false detection; The step of selecting target processing measures according to the missed detection / false detection risk of each image to be detected specifically includes: If there is no missed detection or false detection in the current detection image, a prompt indicating that the detection is normal is sent to the detection feedback interface; If the current detection image has a low risk of missed detection / false detection, an instruction to continue sampling detection is sent to the detection execution end, and a prompt that the detection is continuing is sent to the detection feedback interface; If the current detection image has a high risk of missed detection / false detection, a prompt for manual verification is sent to the detection feedback interface.

8. An object detection device, characterized in that: include: A detection video acquisition module is used to acquire the video clip to be detected; A to-be-detected image extraction module, used to extract a target number of to-be-detected images from the to-be-detected video clip; A detection image input module, used to input the target number of images to be detected into the first detection model and the second detection model respectively in sequence; An integrity test result acquisition module, used to acquire the integrity test result output by the first detection model; and A locality detection result acquisition module, used to obtain the locality detection result output by the second detection model; A verification input module is used to input the overall detection result and the local detection result corresponding to the same image to be detected into a preset consistency verification model; The detection risk prediction module is used to predict the missed detection / false detection risk of each image to be detected based on the preset verification strategy and risk assessment strategy; The processing measure screening module is used to screen out target processing measures according to the missed detection / false detection risk of each image to be detected.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the object detection method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the object detection method according to any one of claims 1 to 7 are implemented.

Images