An object recognition method, device, apparatus and gate

Abstract

The application discloses an object recognition method, device and equipment and a gate, and the method comprises the following steps: acquiring an infrared video stream and a visible light video stream containing a target object; the timing information of the infrared video stream and the visible light video stream is the same; the timing information comprises the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; determining a first target video frame from a plurality of first video frames of the infrared video stream based on the index value information of each first video frame; the index value information is used for representing the state information of the target object in the first video frame; determining a second target video frame corresponding to the first timing corresponding to the first target video frame from a plurality of second video frames of the visible light video stream; and recognizing the target object based on the second target video frame.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and to, but is not limited to, an object recognition method, apparatus, device, and gate. Background Technology

[0002] In related technologies, visible light facial recognition access control systems face the challenge of processing concurrent visible light video streams during terminal operation. Repeated video stream input increases the processing time of the visible light image recognition model, without necessarily improving detection accuracy. Furthermore, poor-quality video streams with environmental interference cannot be effectively detected and filtered, further burdening subsequent processing. Summary of the Invention

[0003] In view of this, embodiments of this application provide an object recognition method, apparatus, device, and gate.

[0004] In a first aspect, embodiments of this application provide an object recognition method, the method comprising: acquiring an infrared video stream and a visible light video stream containing a target object; the infrared video stream and the visible light video stream having the same timing information; the timing information including the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; determining a first target video frame from the plurality of first video frames based on index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information being used to characterize the state information of the target object in the first video frame; determining a second target video frame corresponding to a first timing from the plurality of second video frames of the visible light video stream; and recognizing the target object based on the second target video frame.

[0005] Secondly, embodiments of this application provide an object recognition device, comprising: an acquisition module, configured to acquire an infrared video stream and a visible light video stream containing a target object; the infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; a first determination module, configured to determine a first target video frame from the plurality of first video frames based on index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame; a second determination module, configured to determine a second target video frame corresponding to a first timing from the plurality of second video frames of the visible light video stream; and a first recognition module, configured to recognize the target object based on the second target video frame.

[0006] Thirdly, embodiments of this application provide an electronic device, comprising: an infrared sensing component for acquiring an infrared video stream containing a target object; a visible light sensing component for acquiring a visible light video stream containing the target object; the infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; a processor for determining a first target video frame from the plurality of first video frames based on index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame; the processor is further configured to determine a second target video frame corresponding to a first timing from the plurality of second video frames of the visible light video stream; and identify the target object based on the second target video frame.

[0007] Fourthly, this application provides a turnstile, including an infrared sensing component for acquiring an infrared video stream containing the face of a person when the turnstile is detected; a visible light sensing component for acquiring a visible light video stream containing the face; the infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; a processor for determining a first target video frame from the plurality of first video frames based on the index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame; the processor is further configured to determine a second target video frame corresponding to a first timing from the plurality of second video frames of the visible light video stream; perform face recognition based on the second target video frame; and a controller for controlling the opening and closing of the turnstile based on the result of the face recognition.

[0008] In this embodiment, the target infrared video frame (first target video frame) is first determined from the infrared video stream based on the index value information of the video frame in the infrared video stream used to characterize the state information of the target object. Then, the target visible light video frame (second target video frame) corresponding to the timing of the target infrared video frame is determined from the visible light video stream. By judging the infrared video stream with lower resolution and lower power consumption, the speed of target object recognition for the visible light video stream can be improved and the recognition power consumption can be reduced. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an object recognition method according to an embodiment of this application;

[0010] Figure 2 This is a schematic diagram of an infrared video stream according to an embodiment of this application;

[0011] Figure 3 This is a schematic diagram of a visible light video stream according to an embodiment of this application;

[0012] Figure 4 This is a flowchart illustrating a fast and low-power gate face detection method based on automatic screening of infrared images, according to an embodiment of this application.

[0013] Figure 5 This is a schematic diagram of the composition structure of an object recognition device according to an embodiment of this application;

[0014] Figure 6 This is a schematic diagram of the hardware entity of an electronic device according to an embodiment of this application;

[0015] Figure 7 This is a schematic diagram of the hardware entity of a gate according to an embodiment of this application. Detailed Implementation

[0016] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0017] Figure 1 This is a flowchart illustrating an object recognition method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0018] Step 102: Obtain an infrared video stream and a visible light video stream containing the target object; the timing information of the infrared video stream and the visible light video stream is the same; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream;

[0019] The infrared video stream can be a video stream acquired by an infrared sensor (e.g., an infrared camera). A fixed-length infrared video stream containing n video frames can be maintained from the infrared camera. The infrared video stream can be represented as S. IR ={i1,i2,...i k ...,i n}, i k It can represent the video frame captured at time k in the infrared video stream, i k The corresponding timing sequence is k; the infrared video stream has a low resolution and the acquisition scenario is relatively simple; such as Figure 2 As shown, the infrared video stream may include infrared video frames acquired from time t1 to time t6, and the infrared video stream may also include frames acquired from time t7 to time t8. 12 Infrared video frames captured in real time.

[0020] The visible light video stream can be a video stream acquired by a visible light sensor (e.g., a visible light camera); the visible light camera can be an RGB camera, and a fixed-length visible light video stream containing n video frames can be maintained from the visible light camera. The visible light video stream can be represented as S. RGB ={r1,r2,...r k ...,r n}, r k It can represent the video frame acquired at time k in the infrared video stream, r k The corresponding timing is k; the visible light video stream has a high resolution and the acquisition scene is complex; such as Figure 3 As shown, the visible light video stream may include visible light video frames acquired from time t1 to time t6, and the visible light video stream may also include frames acquired from time t7 to time t8. 12 Visible light video frames captured in real time.

[0021] The object can be anything that appears within the monitoring range of the infrared sensor and the visible light sensor. The object can be a person, an animal, or an item in the environment such as a table, chair, cup, or tree. The target object can be a human body or an animal, and the human body can be the whole body or parts of the human body (such as the face, torso, limbs, etc.).

[0022] Step 104: Based on the index value information of each of the multiple first video frames in the infrared video stream, determine the first target video frame from the multiple first video frames; the index value information is used to characterize the state information of the target object in the first video frame;

[0023] Wherein, if the target object is a human body, the state information can be the human body's motion state or activity level, such as changes in posture, changes in facial expression, human stillness, high-speed movement, relatively fast movement, slow movement, approaching an infrared sensor or visible light sensor, moving away from an infrared sensor or visible light sensor, rapid approach, slow approach, rapid moving away, slow moving away, etc.; the index value information can be some statistical values ​​about the target object's state information; the first target video frame can be from infrared video stream S IR Among multiple first video frames, the video frames whose index value information meets the conditions can be, for example, video frame i. k .

[0024] Because infrared images, compared to visible light images, do not strongly reflect the surrounding environment of a scene, but strongly reflect the human body, they make it easier to distinguish between the user and the surrounding environment. Therefore, it is easier to obtain some indicator values ​​about the user's state than visible light images.

[0025] Step 106: Determine the second target video frame corresponding to the first target video frame in the first time sequence from the multiple second video frames of the visible light video stream;

[0026] Wherein, the first target video frame i k If the corresponding first time sequence is k, then the second target video frame can be a visible light video stream S. RGB Video frames r with time sequence k k .

[0027] Step 108: Identify the target object based on the second target video frame.

[0028] Among them, the second target video frame r k The data is input into the visible light detection module to identify the target object in the second target video frame. The visible light detection module can also be a visible light image recognition module. When the target object is a face, the visible light detection module can be a face recognition model that can perform face recognition on the acquired second target video frame. The face recognition model can first perform image preprocessing on the second target video frame, such as light compensation, grayscale transformation, histogram equalization, normalization, geometric correction, filtering, and sharpening. Then, it can extract facial image features such as eyes, nose, mouth, and chin. Finally, it can compare the extracted facial image features with the facial feature templates stored in the database and determine the identity information of the face based on the degree of similarity.

[0029] In some embodiments, during the concurrent face recognition phase of the terminal visible light face recognition access control system, an infrared video stream is collected by an infrared sensor and used as a "shadow" to follow the acquisition of visible light. During this process, the visible light sensor acquires a visible light video stream with the same number of frames as the infrared video stream, but these video streams may not be processed. Instead, the repeatability and quality of the video frames are judged using the infrared video stream to obtain control information. The control information may be the first timing sequence of a first target video frame that meets the conditions. After receiving the control information transmitted by the infrared sensor, the target visible light video frame is determined based on the control information, and then face recognition processing is performed on the target visible light video frame.

[0030] In this embodiment, the target infrared video frame (first target video frame) is first determined from the infrared video stream based on the index value information of the video frame in the infrared video stream used to characterize the state information of the target object. Then, the target visible light video frame (second target video frame) corresponding to the time sequence of the target infrared video frame is determined from the visible light video stream. By judging the infrared video stream with lower resolution, lower power consumption, and easier differentiation between the human body and the environment, the speed and accuracy of target object recognition for the visible light video stream can be improved, and the power consumption of recognition can be reduced.

[0031] In some embodiments, step 104, "determining the first target video frame from the plurality of first video frames based on the index value information of each of the plurality of first video frames in the infrared video stream," can be implemented through the following steps 141 to 143:

[0032] Step 141: Based on the index value information of each of the multiple first video frames in the infrared video stream, determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames;

[0033] In some embodiments, step 141 can be implemented by the following steps 1411 to 1413:

[0034] Step 1411: Obtain the indicator value information of each of the first video frames to obtain an indicator value sequence; wherein, the corresponding indicator value information can be obtained by mapping the image of each of the first video frames in the infrared video stream through the measurement function f, and multiple indicator value information can generate an indicator value sequence, for example, the first video frame i n-1 The corresponding indicator value information can be represented as p n-1 The first video frame i n The corresponding indicator value information can be represented as p n p n =f(i n The sequence of index values ​​can be represented as s. sup s sup ={p1,p2,...,p n}

[0035] Step 1412: Based on multiple index value information in the index value sequence, generate a fitting curve, determine the temporal change rate of the fitting curve with respect to the first video frame, and obtain a change quantity sequence containing multiple temporal change information;

[0036] Among them, the index value sequence s can be used. supThe elements (i.e., index value information) in the data are subjected to curve fitting to generate a fitted curve. The derivative of the fitted curve with respect to time t can be calculated and quantified to obtain the temporal change rate of the fitted curve with respect to the first video frame. The temporal change rate can be expressed as d. n , This allows us to obtain a change sequence S containing multiple time-series change information. D S D ={d1,d2,...d n The sequence of changes S D It can be stored in the "shadow" register.

[0037] Step 1413: Based on the multiple temporal change information in the change sequence, use a Gaussian function to determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames;

[0038] Step 142: Based on the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames, obtain a priority sequence containing priority scores of the plurality of first video frames;

[0039] Specifically, based on temporal change information, the image clarity of infrared video frames and / or the degree of difference between adjacent infrared video frames can be determined. Based on the image clarity of infrared video frames and / or the degree of difference between adjacent infrared video frames, infrared video frames can be divided into several priority levels to indicate the activity level of the target object. For example, the activity level of the target object can be indicated by determining the priority score of infrared video frames.

[0040] When the temporal change rate is too small, it indicates that the difference between adjacent first video frames is too small, suggesting high repetition of infrared video frames and almost no change in the state of the target object, resulting in a low priority score. When the temporal change rate is too large, it indicates that the difference between adjacent first video frames is too large, suggesting drastic changes in the state of the target object in the infrared video frames, which may result in blurring, also resulting in a low priority score. When the temporal change rate is moderate, the difference is moderate, indicating that the state changes of the target object are appropriate and the quality of the infrared video frames is good, resulting in a high priority score.

[0041] Images with poor clarity, i.e., excessively blurry, receive a lower priority score, while images with relatively clear clarity receive a higher priority score. Excessive blurriness may be caused by factors such as insufficient current, improper circuit design, or harsh environments. It could also be due to drastic changes in the state of the target object, such as high-speed movement of the target object causing video frame blurring.

[0042] Video frames in a video stream can be prioritized based on image clarity and the degree of difference between adjacent video frames, either individually or in combination. Video frames with higher image clarity and moderate changes in the state of the target object can be given higher priority scores, while video frames with lower image clarity and overly gentle or overly dramatic changes in the state of the target object can be given lower priority scores.

[0043] The priority score, for the time-series rate of change, can be a normal function (i.e., a Gaussian function). When setting the mean of the normal function, it can be treated as a probability function, thus performing probability sampling. The probability is higher for values ​​in the middle and lower for values ​​at both ends, avoiding the problem that the actual threshold might be at either end. Regarding the update of the "shadow," the shadow is stored as a window, still using the priority principle of the normal function, with values ​​at both ends being discarded first.

[0044] Assume that the Gaussian function with a given priority follows Using the Gaussian function, priority scoring can be performed on video frames at different time intervals to obtain a priority sequence S. G ={g1,g2,...g n Using this priority sequence, the higher the priority score, the easier it is to be sampled. Therefore, the probability of selecting the video frame corresponding to the nth time sequence is... The probability of discarding the video frame corresponding to the nth time sequence is:

[0045] Step 143: Determine the first video frame in the priority sequence that meets the priority score requirements as the first target video frame.

[0046] The priority score indicates the priority order of the first video frames in the infrared video stream. The first video frame that meets the priority score requirement can be the first video frame with the highest priority score, the first video frame with a priority score greater than a first score threshold, or the first video frame with a priority score greater than a second score threshold and the highest priority score. By assigning a higher priority score to the first target video frame with good quality and a derivative value in the middle, and sending the second target video frame corresponding to the first target video frame into the image recognition model for inference, the speed of image recognition can be accelerated.

[0047] In this embodiment, the temporal change rate of the first video frame can be determined based on the index value information of the first video frame. The image clarity of the first video frame and / or the degree of difference between adjacent first video frames can be determined based on the temporal change information. Based on the image clarity and the degree of difference, the priority score of the first video frame can be determined. In this way, the first target video frame with high image clarity and moderate state change can be determined, thereby improving the speed and accuracy of image recognition.

[0048] In some embodiments, step 1411, “obtaining the index value information for each of the first video frames”, may include obtaining at least one of the image compression ratio of the first video frame and the outline size of the target object in the first video frame.

[0049] The measurement function f can be used to calculate at least one of the image compression ratio and contour size of the first video frame; the image compression ratio can also be called the compression measurement value.

[0050] An image's compression ratio can be understood as the compression of image information. For infrared video frames, the environmental information is mostly the same and is greatly compressed, while the user's information is relatively increased. Therefore, changes in the image's compression ratio can flexibly reflect changes in the user's state. However, for a visible light video frame, the image compression ratio remains consistently high, and even if the person's state changes, the change in the compression ratio is very small. Therefore, changes in the compression ratio of an infrared video frame can largely reflect changes in a person's state.

[0051] In addition, the outline size can be the perimeter of the face outline. The outline size can also reflect changes in the person's state. When the outline size decreases, it can be determined that the person is moving away from the infrared sensor or visible light sensor. The rate at which the outline size decreases can be determined that the person is moving away. When the outline size increases, it can be determined that the person is moving closer to the infrared sensor or visible light sensor. The rate at which the outline size increases can be determined that the person is moving closer.

[0052] When the indicator value information includes multiple statistical values ​​such as the image compression ratio and the outline size of the target object, the normal function can be transformed into a high-dimensional normal density function. Probability sampling similar to that of the normal function can be performed from this high-dimensional normal density function, thereby enabling more accurate sampling of the first target video frame.

[0053] In this embodiment of the application, by obtaining indicator value information including at least one of image compression ratio and contour size, the diversity of the obtained indicator value information is improved.

[0054] In some embodiments, the method further includes steps 110 and 112:

[0055] Step 110: If the target object is successfully identified, update the mean of the Gaussian function so that the updated mean is shifted toward the direction of the first target video frame;

[0056] Step 112: Use the updated Gaussian function to determine the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames;

[0057] Among them, the Gaussian function after updating the mean has a stronger ability to determine the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames than the Gaussian function before updating the mean.

[0058] If the target object is successfully identified after the first target video frame is sampled, a feedback mechanism can be initiated, causing the mean of the Gaussian function to shift towards the sampled point. This allows the image recognition model to continuously converge to a mean center. Assume the temporal sequence of the first target video frame is k, and the priority score corresponding to the first target video frame is g. k The mean of the Gaussian function before the update is μ, and the mean after the update is μ'. The priority score can be differiated from the mean before the update and multiplied by the step size coefficient l to obtain a certain bias, thus yielding the updated mean: μ' = μ - l * (μ - g) k ).

[0059] In this embodiment of the application, when the target object is successfully identified, the mean of the Gaussian function can be updated so that the updated mean is shifted toward the first target video frame. This allows the first target video frame that meets the requirements to be determined more accurately through positive feedback.

[0060] In some embodiments, the method further includes performing steps 114 to 120 until the number of times the target object identification fails reaches the first count:

[0061] Step 114: If the target object identification fails, discard the first target video frame and add the next time-series video frame to the infrared video stream to update the infrared video stream;

[0062] Step 116: Based on the index value information of each of the multiple first video frames in the updated infrared video stream, determine the third target video frame from the multiple first video frames;

[0063] Step 118: Determine the fourth target video frame corresponding to the second time sequence of the third target video frame from the multiple second video frames of the visible light video stream;

[0064] Step 120: Identify the target object based on the fourth target video frame.

[0065] The method for determining the third target video frame from the first video frame in step 116 is similar to the method for determining the first target video frame from the first video frame in step 104; the method for identifying the target object based on the fourth target video frame in step 120 is similar to the method for identifying the target object based on the second target video frame in step 108, and will not be described again here.

[0066] In some embodiments, the method further includes steps 122 and 124:

[0067] Step 122: If the number of times the target object recognition fails reaches the first count, update the variance of the Gaussian function;

[0068] Step 124: Use the updated Gaussian function to determine the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames;

[0069] The Gaussian function after variance update is less effective at determining the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames than the Gaussian function before variance update.

[0070] Specifically, the number of consecutively discarded video frames (c) or the number of times the target object recognition failed can be counted. The number C or the number of times is used to dynamically update the variance σ of the Gaussian function. 2 If the quantity C is large, or the number of times the ingredients are counted reaches the first count, it indicates that the Gaussian function has a relatively small effect on filtering infrared video frames, and σ can be updated. 2 Move in the direction of increase, giving more decision-making power to the image recognition module.

[0071] In this embodiment, when there are many failures in identifying the target object, the variance of the Gaussian function can be updated, giving more decision-making power to the image recognition module and further improving the accuracy of image recognition.

[0072] In related technologies, methods for processing and detecting repetitive interference in visible light video streams generally involve: firstly, using sound-activated sensors and ranging sensors for initial facial target perception. These methods control system time and power consumption before and after generating concurrent video streams, but cannot provide effective decision-making during video stream occurrence. Secondly, for low-quality visible light images, a balance is struck between power consumption and data rate. However, due to the complexity of visible light scenes, such decision-making is greatly affected by the surrounding environment, leading to an increased false positive rate. Therefore, this invention combines the low resolution and simple acquisition scenarios of infrared video streams to dynamically assist in the decision-making and judgment of filtering repetitive interference in visible light video streams, thus proposing a fast and low-power gate face detection method based on infrared camera intervention.

[0073] Figure 4 This is a flowchart illustrating a fast and low-power gate face detection method based on automatic screening using infrared images, as described in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0074] Step 401: Acquire the infrared video stream;

[0075] Among these, a fixed-length infrared video stream S can be maintained from the infrared camera. IR ={i1,i2,...i k ...,i n};

[0076] Step 402: Obtain the visible light video stream;

[0077] One of these features is the ability to maintain a fixed-length visible light video stream from a visible light camera.

[0078] S RGB ={r1,r2,...r k ...,r n};

[0079] Step 403: Map the video frames in the infrared video stream set to obtain the infrared index value sequence by using the measurement function f;

[0080] The infrared index value sequence, also known as the index value sequence, can be represented as s. sup ={p1,p2,...,p n}, where p n =f(i n );

[0081] Step 404: Fit curves to the elements in the infrared index value sequence, and take the derivative of the fitted curve with respect to time to obtain the infrared change sequence;

[0082] The infrared change sequence, also known as the change sequence, is derived with respect to the index value information: This yields a set of time-series change information, namely the infrared change sequence S. D ={d1,d2,...d n};

[0083] Step 405: Define the Gaussian function;

[0084] Step 406: Determine the infrared priority score sequence using the Gaussian function;

[0085] The infrared priority score sequence, also known as the priority sequence, is assumed to be based on a Gaussian function that determines the priority score. Using a Gaussian function, priority scoring is performed on video frames at different time intervals to obtain a priority scoring sequence S. G ={g1,g2,...g n Using a priority scoring sequence, the higher the score, the easier it is for the video frame to be sampled. Therefore, the probability of selecting the video frame corresponding to the nth time sequence is... The probability of discarding the video frame corresponding to the nth time sequence is:

[0086]

[0087] Step 407: Based on the priority score of each video frame in the infrared priority score sequence, determine the first target video frame and the k-th time sequence corresponding to the first target video frame;

[0088] Step 408: Sample the second target video frame corresponding to the k-th time sequence from the visible light video stream;

[0089] Specifically, based on the sampled first target video frame at time k, the second target video frame r at time k in the visible light video stream is found. k The second target video frame is then sent to the visible light image recognition module.

[0090] Step 409: Input the second target video frame into the visible light image recognition module;

[0091] Step 410: Determine if the recognition was successful? If yes, proceed to step 411; otherwise, proceed to step 412.

[0092] Step 411: Update the mean;

[0093] Step 412: Count the number of consecutively discarded frames, c;

[0094] Step 413: Update variance.

[0095] In the output of the visible light detection module, two scenarios can occur. One is that the user's face is correctly identified because of the first target video frame. In this case, k can be selected based on the timing of this action to update the mean μ in the Gaussian function. Let's assume the priority score of the first target video frame is g. k The difference between μ and μ is multiplied by the step size coefficient l to obtain a certain bias and thus the updated mean, μ' = μ - l * (μ - g) k ).

[0096] In another scenario, if the first target video frame fails to correctly identify the user's face, it is discarded, and the video stream window enters a new time frame. The operations S2 to S4 described above are performed on this time frame to obtain a window of fixed length for selection. The number of consecutively discarded frames, C, is counted and used to dynamically update σ. If the number is large, it indicates that the method for determining the first target video frame is ineffective for the current infrared video stream. The update σ moves in the direction of increasing, giving more decision-making power to the subsequent visible light detection module.

[0097] In this embodiment, the detection speed of the visible light system and the detection power consumption can be improved by judging the infrared data stream on a terminal device with limited computing power.

[0098] It should be noted that, in the embodiments of this application, if the above-described object recognition method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, desktop computer, personal digital assistant, navigator, digital phone, video phone, television, sensor device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0099] Figure 5 This is a schematic diagram of the composition structure of an object recognition device according to an embodiment of this application, as shown below. Figure 5 As shown, the device 500 includes: an acquisition module 501, a first determination module 502, a second determination module 503, and a first identification module 504, wherein:

[0100] The acquisition module 501 is used to acquire an infrared video stream and a visible light video stream containing the target object; the timing information of the infrared video stream and the visible light video stream is the same; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream;

[0101] The first determining module 502 is used to determine a first target video frame from the plurality of first video frames based on the index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame;

[0102] The second determining module 503 is used to determine, from the plurality of second video frames of the visible light video stream, the second target video frame corresponding to the first target video frame in the first time sequence;

[0103] The first identification module 504 is used to identify the target object based on the second target video frame.

[0104] In some embodiments, the first determining module 502 includes: a first determining submodule, configured to determine the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames based on the index value information of each of the plurality of first video frames in the infrared video stream; a second determining submodule, configured to obtain a priority sequence containing priority scores of the plurality of first video frames based on the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames; and a third determining submodule, configured to determine the first video frame in the priority sequence that meets the priority score requirement as the first target video frame.

[0105] In some embodiments, the first determining submodule includes: a first acquiring unit, configured to acquire indicator value information for each of the first video frames to obtain an indicator value sequence; a first determining unit, configured to generate a fitting curve based on multiple indicator value information in the indicator value sequence, determine the temporal change rate of the fitting curve with respect to the first video frame, and obtain a sequence containing multiple temporal change information; and a second determining unit, configured to determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames using a Gaussian function based on the multiple temporal change information in the sequence.

[0106] In some embodiments, the first acquisition unit is configured to acquire at least one of the image compression ratio of the first video frame and the outline size of the target object in the first video frame.

[0107] In some embodiments, the apparatus further includes: a first updating module, configured to update the mean of the Gaussian function when the target object is successfully identified, such that the updated mean is shifted toward the direction of the first target video frame; and a third determining module, configured to determine the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames using the Gaussian function after updating the mean; wherein the ability of the Gaussian function after updating the mean to determine the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames is stronger than that of the Gaussian function before updating the mean.

[0108] In some embodiments, the apparatus further includes: a discarding module, configured to perform the following steps until the number of times the target object identification fails reaches a first count: in the case of failure to identify the target object, discarding the first target video frame and adding a video frame of the next time sequence to the infrared video stream to update the infrared video stream; a fourth determining module, configured to determine a third target video frame from the plurality of first video frames based on the index value information of each of the first video frames in the updated infrared video stream; a fifth determining module, configured to determine a fourth target video frame corresponding to the second time sequence of the third target video frame from the plurality of second video frames in the visible light video stream; and a second identification module, configured to identify the target object based on the fourth target video frame.

[0109] In some embodiments, the apparatus further includes: a second updating module, configured to update the variance of the Gaussian function when the number of times the target object recognition fails reaches the first count; and a sixth determining module, configured to determine the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames using the Gaussian function after the variance update; wherein the ability of the Gaussian function after the variance update to determine the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames is weaker than that of the Gaussian function before the variance update.

[0110] Correspondingly, embodiments of this application provide an electronic device, Figure 6 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application, such as... Figure 6As shown, the hardware entity of the electronic device 600 includes: an infrared sensing component 601 for acquiring an infrared video stream containing a target object; a visible light sensing component 602 for acquiring a visible light video stream containing the target object; the infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; a processor 603 for determining a first target video frame from the plurality of first video frames based on the index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame; the processor 603 is also used to determine a second target video frame corresponding to the first timing of the first target video frame from the plurality of second video frames of the visible light video stream; and to identify the target object based on the second target video frame.

[0111] Figure 7 This is a schematic diagram of the hardware entity of a gate according to an embodiment of this application, as shown below. Figure 7 As shown, the turnstile includes an infrared sensing component 701, used to acquire an infrared video stream containing the face of a person when the turnstile is detected; a visible light sensing component 702, used to acquire a visible light video stream containing the face; the timing information of the infrared video stream and the visible light video stream is the same; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; a processor 703, used to determine a first target video frame from the plurality of first video frames based on the index value information of each first video frame in the plurality of first video frames of the infrared video stream; the index value information is used to characterize the state information of the target object in the first video frame; the processor 703 is also used to determine a second target video frame corresponding to the first timing of the first target video frame from the plurality of second video frames of the visible light video stream; and to perform face recognition based on the second target video frame; a controller 704, used to control the opening and closing of the turnstile based on the result of the face recognition.

[0112] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the object recognition method provided in the above embodiments.

[0113] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects as the device embodiments. For technical details not disclosed in the storage medium and method embodiments of this application, please refer to the descriptions of the device embodiments of this application for understanding.

[0114] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this application may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0118] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause computer devices (which may be mobile phones, tablets, desktops, personal digital assistants, navigators, digital phones, video phones, televisions, sensing devices, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0119] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments without conflict.

[0120] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An object recognition method, the method comprising: Acquire infrared and visible light video streams containing the target object; The infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing of each video frame in the video stream and the number of video frames in the video stream. The infrared video stream serves as a shadow to follow the acquisition of the visible light video stream, and control information is obtained through the infrared video stream to identify and process the visible light video stream. Based on the index value information of each first video frame in the multiple first video frames of the infrared video stream, a fitting curve is generated, and multiple time-series change information is determined according to the fitting curve. Based on the multiple temporal change information, the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames are determined using a Gaussian function; the index value information is used to characterize the state information of the target object in the first video frame. Based on the image clarity of the plurality of first video frames and / or the degree of difference between adjacent first video frames, a priority sequence containing priority scores of the plurality of first video frames is obtained; The first video frame in the priority sequence that meets the priority score requirement is determined as the first target video frame; The second target video frame corresponding to the first target video frame in the first time sequence is determined from multiple second video frames of the visible light video stream; The target object is identified based on the second target video frame.

2. The method according to claim 1, wherein, The method involves generating a fitting curve based on the index value information of each of the multiple first video frames in the infrared video stream, and determining multiple time-series change information based on the fitting curve, including: Obtain the indicator value information for each of the first video frames to obtain the indicator value sequence; A fitting curve is generated based on multiple indicator value information in the indicator value sequence; The temporal rate of change of the fitted curve with respect to the first video frame is determined to obtain a sequence of changes containing multiple temporal change information.

3. The method according to claim 2, wherein, The step of obtaining the indicator value information for each of the first video frames includes: Obtain at least one of the image compression ratio of the first video frame and the outline size of the target object in the first video frame.

4. The method according to claim 2, wherein, The method further includes: If the target object is successfully identified, the mean of the Gaussian function is updated so that the updated mean is shifted toward the direction of the first target video frame. The image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames are determined using a Gaussian function with updated mean. Among them, the Gaussian function after updating the mean has a stronger ability to determine the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames than the Gaussian function before updating the mean.

5. The method according to claim 2, wherein, The method further includes: Perform the following steps until the number of times the target object identification fails reaches the first count: If the target object fails to be identified, the first target video frame is discarded, and the next time-series video frame is added to the infrared video stream to update the infrared video stream. Based on the index value information of each of the multiple first video frames in the updated infrared video stream, a third target video frame is determined from the multiple first video frames. The fourth target video frame corresponding to the second time sequence of the third target video frame is determined from multiple second video frames of the visible light video stream; The target object is identified based on the fourth target video frame.

6. The method according to claim 5, wherein, The method further includes: If the number of times the target object identification fails reaches the first count, update the variance of the Gaussian function; The image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames are determined by using a Gaussian function after updating the variance. The Gaussian function after variance update is less effective at determining the image sharpness of the plurality of first video frames and / or the degree of difference between adjacent first video frames than the Gaussian function before variance update.

7. An object recognition device, the device comprising: The acquisition module is used to acquire infrared video streams and visible light video streams containing the target object; The timing information of the infrared video stream and the visible light video stream is the same; the timing information includes the timing of each video frame in the video stream and the number of video frames in the video stream; wherein, the infrared video stream is used as a shadow to follow the acquisition of the visible light video stream, and control information is obtained through the infrared video stream to identify and process the visible light video stream. The first determining module is configured to generate a fitting curve based on the index value information of each of the multiple first video frames in the infrared video stream, and determine multiple temporal change information based on the fitting curve; based on the multiple temporal change information, use a Gaussian function to determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames; the index value information is used to characterize the state information of the target object in the first video frame; based on the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames, obtain a priority sequence containing priority scores of multiple first video frames; and determine the first video frame in the priority sequence that meets the priority score requirements as the first target video frame. The second determining module is used to determine, from the plurality of second video frames of the visible light video stream, the second target video frame corresponding to the first target video frame in the first time sequence; The first identification module is used to identify the target object based on the second target video frame.

8. An electronic device, the device comprising: Infrared sensing components are used to acquire infrared video streams containing target objects; Visible light sensing component, used to acquire visible light video stream containing the target object; The timing information of the infrared video stream and the visible light video stream is the same; the timing information includes the timing of each video frame in the video stream and the number of video frames in the video stream; wherein, the infrared video stream is used as a shadow to follow the acquisition of the visible light video stream, and control information is obtained through the infrared video stream to identify and process the visible light video stream. The processor is configured to generate a fitting curve based on the index value information of each of the multiple first video frames in the infrared video stream, and determine multiple temporal change information based on the fitting curve; determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames using a Gaussian function based on the multiple temporal change information; the index value information is used to characterize the state information of the target object in the first video frame; obtain a priority sequence containing priority scores of multiple first video frames based on the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames; and determine the first video frame in the priority sequence that meets the priority score requirements as the first target video frame. The processor is further configured to determine, from a plurality of second video frames in the visible light video stream, a second target video frame corresponding to a first time sequence corresponding to the first target video frame; and to identify the target object based on the second target video frame.

9. A turnstile, the turnstile comprising: An infrared sensing component is used to detect when a person passes through the gate and capture an infrared video stream containing the person's face; A visible light sensing component is used to acquire a visible light video stream containing the face; the infrared video stream and the visible light video stream have the same timing information; the timing information includes the timing corresponding to each video frame in the video stream and the number of video frames in the video stream; wherein, the infrared video stream acts as a shadow to follow the acquisition of the visible light video stream, and control information is obtained through the infrared video stream to perform identification processing on the visible light video stream; The processor is configured to generate a fitting curve based on the index value information of each of the multiple first video frames in the infrared video stream, and determine multiple temporal change information based on the fitting curve; determine the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames using a Gaussian function based on the multiple temporal change information; the index value information is used to characterize the state information of the target object in the first video frame; obtain a priority sequence containing priority scores of multiple first video frames based on the image clarity of the multiple first video frames and / or the degree of difference between adjacent first video frames; and determine the first video frame in the priority sequence that meets the priority score requirements as the first target video frame. The processor is further configured to determine, from a plurality of second video frames in the visible light video stream, a second target video frame corresponding to the first target video frame in a first time sequence; and to perform face recognition based on the second target video frame; A controller is used to control the opening and closing of the gate based on the result of the facial recognition.

Images