Target detection method and apparatus, and electronic device, computer-readable storage medium and computer program product

Abstract

Provided in the present application are a target detection method and apparatus, and an electronic device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a current image frame, wherein the current image frame is a video frame image collected at the current moment; identifying at least one detection box from the current image frame, wherein each detection box comprises an object; determining the detection state of a target object in the current image frame, wherein the detection state indicates whether the target object is lost in the current image frame, and the target object is an object to be detected among at least one object; and on the basis of the detection state, and on the basis of motion information of the target object and a pre-stored appearance feature of the target object, or on the basis of the pre-stored appearance feature of the target object, determining from among the at least one detection box a target detection box comprising the target object. The present application can effectively achieve a balance between the detection speed and the detection effect of an algorithm.

Description

Target detection method and device, electronic device, computer readable storage medium and computer program product

[0001] This application claims priority to the Chinese patent application No. 202411020993.9, filed on July 26, 2024 in the China Patent Office and entitled "Target detection method, device, electronic device, computer readable storage medium and computer program product", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a target detection method, device, electronic device, computer readable storage medium and computer program product. BACKGROUND

[0003] Two-dimensional multi-target tracking algorithms are divided into a method based on motion information matching and a method based on motion information and appearance information matching according to whether appearance information is used, wherein the method based on motion information matching cannot realize the recovery of the target after the human body walks out of the picture (i.e., the detection effect is poor), and when the method based on motion information and appearance information is deployed to a low-power end-side robot, the overall calculation speed of the algorithm will be slow, thereby causing the target to be easily lost. That is, the related art cannot achieve a balance between the detection speed and the detection effect of the algorithm. TECHNICAL PROBLEM

[0004] The embodiments of the present application provide a target detection method, device, electronic device, computer readable storage medium and computer program product, which can effectively balance between the detection speed and the detection effect of the algorithm. TECHNICAL SOLUTION

[0005] The technical solution of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a target detection method, comprising:

[0007] obtaining a current frame image, wherein the current frame image is a video frame image collected at a current time;

[0008] identifying at least one detection box from the current frame image, wherein each detection box includes an object;

[0009] determining a detection state of a target object in the current frame image, wherein the detection state represents whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object;

[0010] According to the detection state, a target detection frame including the target object is determined from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object.

[0011] The embodiment of the application provides a target detection device, comprising:

[0012] An acquisition module is configured to acquire a current frame image, wherein the current frame image is a video frame image acquired at a current time;

[0013] An identification module is configured to identify at least one detection frame from the current frame image, wherein each detection frame includes an object;

[0014] A determination module is configured to determine a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object.

[0015] The determination module is further configured to determine a target detection frame including the target object from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object according to the detection state.

[0016] In the above scheme, the determination module is further configured to determine a target detection frame including the target object from the at least one detection frame based on the motion information of the target object and the pre-stored appearance features of the target object when the detection state indicates that the target object is not lost in the current frame image.

[0017] In the above scheme, the determination module is further configured to determine a detection frame matching the motion information of the target object from the at least one detection frame; when the number of the matching detection frames is one, extract features of an object included in the matching detection frame to obtain appearance features of the object; determine a similarity between the appearance features of the object and pre-stored appearance features of the target object in a feature queue; and when the similarity is greater than a first similarity threshold, determine the matching detection frame as the target detection frame including the target object.

[0018] In the scheme, the feature queue stores N appearance features of the target object in the previous N frames of images respectively, where N is a positive integer greater than 1. The determining module is further configured to transpose a feature vector corresponding to the appearance feature of the object to obtain a transposed feature vector; multiply the transposed feature vector with feature vectors corresponding to the N appearance features respectively to obtain N first multiplication results; subtract a preset value from the N first multiplication results respectively to obtain N first subtraction results; and take the minimum value in the N first subtraction results as a similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue.

[0019] In the scheme, the determining module is further configured to perform position prediction on the target object based on motion information of the target object to obtain a predicted position of the target object in the current frame of image; determine distances between the at least one detection frame and the predicted position respectively; and take a detection frame with a distance less than a distance threshold as a detection frame matched with the motion information of the target object.

[0020] In the scheme, the determining module is further configured to obtain motion information of the target object, where the motion information includes at least one of the following: a horizontal pixel position and a horizontal direction velocity of a tracking frame center associated with the target object, a vertical pixel position and a vertical direction velocity of the tracking frame center associated with the target object, an area and an area change velocity of the tracking frame associated with the target object, and an aspect ratio of the tracking frame associated with the target object; construct a state model based on the motion information; and call the state model to perform position prediction on the target object to obtain a predicted position of the target object in the current frame of image.

[0021] In the scheme, the determining module is further configured to subtract the position of the detection frame from the predicted position to obtain a second subtraction result; transpose the second subtraction result to obtain a transposed second subtraction result; obtain a covariance matrix between the position of the detection frame and the predicted position; multiply the transposed second subtraction result with an inverse matrix of the covariance matrix to obtain a second multiplication result; multiply the second multiplication result with the second subtraction result, and take the obtained multiplication result as a distance between the detection frame and the predicted position.

[0022] In the scheme, the determining module is further configured to determine a detection frame matching the motion information of the target object from the at least one detection frame; when the number of the matching detection frames is more than one, perform feature extraction on objects included in the matching detection frames respectively to obtain appearance features corresponding to the objects respectively; determine the similarity between the appearance feature of each object and the appearance feature of the target object pre-stored in the feature queue respectively; and when the number of the similarities greater than a second similarity threshold is one, take the detection frame corresponding to the similarity greater than the second similarity threshold as the target detection frame including the target object.

[0023] In the scheme, the determining module is further configured to, when the number of the similarities greater than the second similarity threshold is more than one, and the number of the similarities greater than a third similarity threshold in the similarities greater than the second similarity threshold is one, take the detection frame corresponding to the similarity greater than the third similarity threshold as the target detection frame including the target object, where the third similarity threshold is greater than the second similarity threshold.

[0024] In the scheme, the determining module is further configured to, when the detection state indicates that the target object is lost in the current frame image, determine a target detection frame including the target object from the at least one detection frame based on the appearance feature of the target object pre-stored.

[0025] In the scheme, the determining module is further configured to perform feature extraction on objects included in the at least one detection frame respectively to obtain appearance features corresponding to the objects respectively; determine the similarity between the appearance feature of each object and the appearance feature of the target object pre-stored respectively; when the number of the similarities greater than a fourth similarity threshold is one, take the detection frame corresponding to the similarity greater than the fourth similarity threshold as the target detection frame including the target object; and when the number of the similarities greater than the fourth similarity threshold is more than one, and the number of the similarities greater than a fifth similarity threshold in the similarities greater than the fourth similarity threshold is one, take the detection frame corresponding to the similarity greater than the fifth similarity threshold as the target detection frame including the target object, where the fifth similarity threshold is greater than the fourth similarity threshold.

[0026] In the scheme, the appearance feature of the target object is stored in a pre-created feature queue, and the device further comprises a storage module configured to extract features of the target object included in the target detection frame to obtain the latest appearance feature of the target object; when the number of the appearance features of the target object stored in the feature queue is less than a number threshold, the latest appearance feature is inserted into the tail of the feature queue; when the number of the appearance features of the target object stored in the feature queue is equal to the number threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted into the tail of the feature queue.

[0027] In the scheme, the determination module is further configured to obtain an association state between a tracker associated with the target object and the detection frame; when the association state indicates that the tracker has an associated detection frame, or when a time length during which the tracker does not have an associated detection frame is less than a time length threshold, it is determined that the target object is not lost in the current frame image; when the time length during which the tracker does not have an associated detection frame is greater than the time length threshold, it is determined that the target object is lost in the current frame image.

[0028] An electronic device is provided in an embodiment of the present application, and the electronic device comprises:

[0029] A memory is configured to store executable instructions.

[0030] A processor is configured to execute the executable instructions stored in the memory, so as to implement a target detection method provided in an embodiment of the present application.

[0031] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores computer executable instructions, which are configured to be executed by a processor, so as to implement a target detection method provided in an embodiment of the present application.

[0032] A computer program product is provided in an embodiment of the present application, and the computer program product comprises a computer program or computer executable instructions, which are configured to be executed by a processor, so as to implement a target detection method provided in an embodiment of the present application. Advantages

[0033] An embodiment of the present application has the following advantages:

[0034] The target detection system 100 provided by the embodiment of the present application comprises a target detection method, and the target detection method comprises the following steps: acquiring a current frame image, identifying at least one detection frame from the current frame image, and determining a detection state of a target object in the current frame image, wherein the detection state represents whether the target object is lost in the current frame image; and finally, according to the detection state, the target detection frame including the target object is determined from the at least one detection frame based on a combination of motion information and appearance features or based on appearance features only. That is, the embodiment of the present application adopts different ways to detect the target object by judging whether the target object is lost in the current frame image, so as to effectively balance the detection speed and the detection effect of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0035] FIG. 1 is a schematic diagram of the architecture of a target detection system 100 provided by the embodiment of the present application;

[0036] FIG. 2 is a schematic diagram of the structure of an electronic device 500 provided by the embodiment of the present application;

[0037] FIG. 3 is a schematic diagram of the flow of a target detection method provided by the embodiment of the present application;

[0038] FIG. 4 is a schematic diagram of the flow of a target detection method provided by the embodiment of the present application;

[0039] FIG. 5 is a schematic diagram of the flow of a target detection method provided by the embodiment of the present application;

[0040] FIG. 6 is a schematic diagram of the flow of a target detection method provided by the embodiment of the present application;

[0041] FIG. 7 is a schematic diagram of the flow of a target detection method provided by the embodiment of the present application. Embodiments of the present application

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0043] In the following description, “some embodiments” are related to a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0044] It can be understood that, in the embodiments of the present application, the data related to user information (such as the feature information of the user) is involved, and when the embodiments of the present application are applied to specific products or technologies, the user's permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with relevant laws, regulations and standards.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0046] In the following description, the term "first\second\..." only distinguishes similar objects, and does not represent a specific order of the objects. Understandably, "first\second\..." can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0048] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0049] 1) Tracker: Tracker is an algorithm or software module used to continuously identify and track one or more targets in video streams or other sensor data.

[0050] 2) Tracking box: In a target tracking algorithm, a tracking box is a tool used to define the area of an object to be tracked in an image. The tracking box is usually a rectangle that detects the position of the object to be tracked in video or image data.

[0051] 3) Detection box: In a target tracking algorithm, a detection box is a tool used to define the area of an object in an image. The detection box is usually a rectangle that detects the position of all objects in video or image data for matching the detected objects with the object to be tracked.

[0052] 4) Feature queue: Feature queue is a data structure used to process image or video data in computer vision, which is essentially a queue that can store feature data extracted from image or video frames; the storage of the feature queue is based on the "first-in, first-out" storage principle, that is, the feature data to be stored is inserted from the tail of the feature queue for storage, and when deleted, the feature data that has been in the feature queue the longest is deleted from the head of the feature queue.

[0053] 5) Intersection over Union (IOU) threshold: IOU threshold is an important parameter in object detection and tracking for evaluating the degree of overlap between two bounding boxes. In the object tracking task, the IOU threshold is used to evaluate the matching degree between the predicted target position and the true target position. When the IOU between the predicted target position and the true target position is greater than the set threshold, it is considered that the two positions are matched, i.e., the predicted target position is correct.

[0054] 6) Cascade matching: Cascade matching is a strategy commonly used in scenarios that require efficient processing of large-scale data or rapid iteration of model prediction results. The core idea is to design a series of simple and fast filtering steps (or models) to filter out unlikely matches at different stages, leaving only the best matching results. Through the ordered hierarchical arrangement, the number of candidate matches that need to be evaluated in detail is greatly reduced, saving the consumption of overall computing resources and improving processing efficiency.

[0055] Target tracking algorithms are divided into single-target tracking algorithms and multi-target tracking algorithms. Single-target tracking algorithms only focus on the target itself during tracking and do not combine information from other human bodies in the image. Therefore, the tracking effect is poor when human bodies cross, and long-term tracking is not possible (i.e., the target cannot be found again after leaving the image). Multi-target tracking algorithms consider the appearance information of all human bodies in the image during tracking. When human bodies cross, the optimal match can be obtained by combining the motion information of multiple human bodies, effectively improving the tracking effect when human bodies cross. Furthermore, by combining the appearance information of human bodies for tracking and matching, the tracking effect when human bodies cross can be further improved, and the target can be found again after the target human body leaves the image.

[0056] Two-dimensional multi-target tracking algorithms are mainly divided into motion information-based matching and motion information and appearance information-based matching methods according to whether appearance information is used.

[0057] When using a motion information-based matching method for tracking, representative algorithms include SORT and ByteTrack, which require the use of a target detection algorithm to detect human bodies in each frame of the image. Although they combine the motion information of multiple human bodies to obtain the optimal match and improve the performance when human bodies cross, ID switching can still occur when the trajectories of human bodies are close, there are detection errors, or there are posture changes. Furthermore, there is no way to find the target after the human body leaves the image.

[0058] When tracking is performed using a classic method based on motion information and appearance information matching, a representative algorithm is Deep SORT, etc. In addition to the need for detection of human bodies in each frame of picture, the algorithm also needs to extract features of all human bodies in the picture, and when matching, it is necessary to simultaneously judge whether the motion information is continuous (whether the motion information meets a threshold) and whether the appearance is similar (whether the similarity meets a threshold), which can further improve tracking errors caused by detection errors, posture changes and other factors in the process of human body crossing, and when the target human body walks out of the picture, the target can be retrieved in combination with the stored appearance information. However, the algorithm needs to perform human body detection and multiple human body feature extraction in each frame, and when the algorithm runs on a low-power end-side robot, the overall calculation speed of the algorithm becomes very slow, which in turn causes the target to be easily lost.

[0059] Based on this, the embodiments of the present application provide a target detection method and device, electronic equipment, computer readable storage medium and computer program product, which can effectively balance the detection speed and detection effect of the algorithm. The electronic equipment provided by the embodiments of the present application can be implemented as a server, or cooperatively implemented by a server and a terminal. The following will take the target detection method cooperatively implemented by the server and the terminal as an example for description.

[0060] For example, referring to FIG. 1, which is an architecture schematic diagram of a target detection system 100 provided by the embodiments of the present application, in order to realize a target detection application, as shown in FIG. 1, the target detection system 100 includes a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 through the network 300, wherein the network 300 can be a local area network or a wide area network, or a combination of the two; the terminal 400 is a terminal associated with a user, and a client 410 is running on the terminal 400. The client 410 can be various types of clients, for example, including a dedicated target detection client, a robot navigation client, etc.

[0061] In some embodiments, the user can collect a video frame image of the current moment through the client 410, the terminal 400 transmits the obtained video frame image to the server 200 through the network 300, then the server 200 identifies at least one detection box from the current frame image, wherein each detection box includes an object; then the server 200 determines the detection state of the target object in the current frame image, wherein the detection state represents whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object; after that, the server 200 can determine the target detection box including the target object from the at least one detection box according to the detection state, based on the motion information of the target object and the pre-stored appearance feature of the target object, or based on the pre-stored appearance feature of the target object, to realize the detection of the target object. Finally, the server 200 sends the target detection box including the target object to the terminal 400 through the network 300, and presents it on the client 410.

[0062] Of course, the above process can also be implemented by the terminal 400 alone, for example, the terminal 400 can determine the target detection box including the target object from the at least one detection box based on its own computing power, and the embodiments of the present application do not make specific limitations in this regard.

[0063] It should be noted that the technical solutions provided by the present application can be applied to various scenarios, such as automatic driving, human-computer interaction, robot navigation, etc. For example, applying the target detection scenario provided by the present application to the robot target tracking scenario with limited computing power on the terminal side can realize stable tracking of a specific object; in game and virtual reality applications, tracking of user gestures or body movements can be realized to provide a more natural and intuitive user interaction experience; in the scenario of an automatic driving car, other vehicles, pedestrians or any potential obstacles on the road can be identified and tracked, and by accurately tracking these targets, the driving path and speed of the vehicle can be better determined to ensure driving safety, and the embodiments of the present application do not make specific limitations on the application scenarios of the target detection method.

[0064] In other embodiments, the embodiments of the present application can also be implemented by means of cloud technology. Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or local area network, to realize the calculation, storage, processing and sharing of data.

[0065] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on cloud computing business model, which can form a resource pool, and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources.

[0066] For example, the server 200 in FIG. 1 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, a robot, and the like, but is not limited thereto. The terminal 400 and the server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0067] The structure of the electronic device provided in the embodiments of the present application will be further described below. Taking the electronic device as a terminal for example, referring to FIG. 2, FIG. 2 is a structural schematic diagram of an electronic device 500 provided in the embodiments of the present application. The electronic device 500 shown in FIG. 2 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in FIG. 2.

[0068] The processor 510 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0069] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0070] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices physically located in proximity to the processor 510.

[0071] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0072] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0073] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0074] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), etc.

[0075] The presentation module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, etc.), such as a user interface for operating peripheral devices and displaying content and information.

[0076] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0077] In some embodiments, the device provided by the embodiments of the present application can be implemented in a software manner, and FIG. 2 shows a target detection device 555 stored in the memory 550, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 5551, an identification module 5552, a determination module 5553, and a storage module 5554. These modules are logical, and thus can be combined or further split according to the functions implemented. It should be noted that in FIG. 2, all the above modules are shown at one time for the convenience of expression, but should not be regarded as excluding the implementation that the target detection device 555 can only include the acquisition module 5551, the identification module 5552, and the determination module 5553. The functions of each module will be described below.

[0078] The target detection method provided by the embodiments of the present application will be described in detail below in combination with the exemplary application and implementation of the terminal device provided by the embodiments of the present application.

[0079] Referring to FIG. 3, FIG. 3 is a flowchart of the target detection method provided by the embodiments of the present application, which will be described in combination with the steps shown in FIG. 3.

[0080] It should be noted that the method shown in FIG. 3 can be executed by various forms of computer programs run by the terminal device, and is not limited to the client, for example, it can also be the operating system, software module, script and applet and the like described above, therefore, the following examples of the client should not be regarded as a limitation of the embodiments of the present application. In addition, for the convenience of description, the terminal and the client run by the terminal are not specifically distinguished in the following.

[0081] In step 101, a current frame image is acquired.

[0082] Here, the current frame image can refer to a video frame image collected at the current time.

[0083] In some embodiments, the image at the current time can be captured in real time by a camera, and the video frame image at the current time can also be collected in real time by a camera or other video shooting device.

[0084] For example, to acquire the current video frame image in the real-time video stream, the cv2.VideoCapture(0) method can be used to acquire the video stream from the camera, and it should be noted that in actual application, it is necessary to ensure the right to access the required video file or camera.

[0085] In step 102, at least one detection box is identified from the current frame image.

[0086] Here, each detection box can include an object.

[0087] In some embodiments, first, the current video frame image is converted into a format suitable for a deep learning model (for example, the video frame image can be resized, normalized, converted to a floating point number, etc.), and image enhancement techniques (such as cropping, rotating, scaling, etc.) can also be used to improve detection effect, to realize the preprocessing of the video frame image; then, a deep learning framework (such as TensorFlow, PyTorch, OpenCV, etc.) is used to load the trained target detection model, to ensure that the model has been trained and can normally run on the given input data; then, the preprocessed video frame image is input into the target detection model for forward propagation calculation, and the model outputs the corresponding prediction result (such as target category, confidence and target bounding box coordinates, etc.); then, the bounding box coordinates, category and confidence of the model prediction result are parsed, since a single object can be detected by multiple bounding boxes, it is necessary to eliminate overlapping bounding boxes to retain the bounding box that is most likely to contain the target object; finally, the corresponding detection box is drawn on the video frame image according to the bounding box, and the drawn detection box is displayed for subsequent target detection.

[0088] In step 103, the detection state of the target object in the current frame image is determined.

[0089] Here, the detection state can be used to represent whether the target object is lost in the current frame image, and the target object is at least one object to be detected (for example, user 1).

[0090] In some embodiments, referring to FIG. 4, FIG. 4 is a flowchart of a target detection method provided by an embodiment of the present application, as shown in FIG. 4, the step 103 shown in FIG. 3 can be implemented by the steps 1031 to 1033 shown in FIG. 4, which will be described in conjunction with the steps shown in FIG. 4.

[0091] In step 1031, the association relationship between the tracker associated with the target object and the detection box is obtained.

[0092] Here, the association relationship can be used to represent whether the tracker associated with the target object (for example, user 1) has an associated detection box.

[0093] In some embodiments, in the target tracking process, the tracker interacts with the detection box to determine and identify the target object to be tracked. When the tracker has an associated detection box, it means that the tracker has a detection box in the current frame image that matches the target object associated with the tracker; when the tracker does not have an associated detection box, it means that the tracker does not have a detection box in the current frame image that matches the target object associated with the tracker.

[0094] In step 1032, when the associated state feature tracker has the associated bounding box, or when the time length that the associated state feature tracker does not have the associated bounding box is less than the time length threshold, it is determined that the target object is not lost in the current frame image.

[0095] Here, when the target object is not lost in the current frame image or the time length of the loss is short, it is considered that the target object is not lost in the current frame image.

[0096] In some embodiments, when the associated state feature tracker has the associated bounding box, it means that the target object is detected and not lost in the current frame image.

[0097] In some embodiments, when the time length that the associated state feature tracker does not have the associated bounding box is less than the time length threshold, it means that the target object is not detected in the current frame image, but the time length of the loss of the target object is short and does not reach the time length threshold, and it is considered that the target object is not lost in the current frame image.

[0098] In step 1033, when the time length that the associated state feature tracker does not have the associated bounding box is greater than the time length threshold, it is determined that the target object is lost in the current frame image.

[0099] In some embodiments, when the time length that the associated state feature tracker does not have the associated bounding box is greater than the time length threshold, it means that the target object has been lost for a long time, and it is considered that the target object has been lost in the current frame image.

[0100] Here, when the target object is lost in the current frame image for a long time, it is considered that the target object has been lost in the current frame image.

[0101] In step 104, according to the detection state, the target detection box including the target object is determined from the at least one bounding box based on the motion information of the target object and the pre-stored appearance feature of the target object, or based on the pre-stored appearance feature of the target object.

[0102] Here, the detection state includes two states that the target object is not lost in the current frame image and the target object is lost in the current frame image.

[0103] In some embodiments, referring to FIG. 5, which is a flowchart of a target detection method provided by an embodiment of the present application, as shown in FIG. 5, step 104 shown in FIG. 3 can be implemented by step 1041A or step 1041B shown in FIG. 5, which will be described in combination with the steps shown in FIG. 5.

[0104] In step 1041A, when the detection state indicates that the target object is not lost in the current frame image, a target detection frame including the target object is determined from the at least one detection frame based on the motion information of the target object and the appearance feature of the target object pre-stored.

[0105] In some embodiments, when the target object is not lost in the current frame image, a target detection frame including the target object can be determined from the at least one detection frame included in the current frame image according to the motion information of the target object and the appearance feature.

[0106] In some embodiments, the step 1041A described above can be implemented in the following manner: determining a detection frame matching the motion information of the target object from the at least one detection frame; when the number of matching detection frames is one, performing feature extraction on the object included in the matching detection frame to obtain the appearance feature of the object; determining the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue; and when the similarity is greater than a first similarity threshold, determining the matching detection frame as the target detection frame including the target object.

[0107] For example, assuming that the first similarity threshold is 0.5 and the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue is 0.7, i.e., greater than the first similarity threshold 0.5, the matching detection frame can be determined as the target detection frame including the target object.

[0108] In some embodiments, when the number of matching detection frames is one, the object included in the matching detection frame is directly subjected to feature extraction to obtain the appearance feature (such as color, texture, shape, edge information, etc.) of the object; in addition, when the dimension of the appearance feature of the object is high, representative features can be selected or the dimension of the appearance feature can be reduced by dimension reduction (such as principal component analysis, etc.).

[0109] In some embodiments, the determination of the detection frame matching the motion information of the target object from the at least one detection frame described above can be implemented in the following manner: performing position prediction on the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image; respectively determining the distance between the at least one detection frame and the predicted position; and taking the detection frame with a distance less than a distance threshold as the detection frame matching the motion information of the target object.

[0110] For example, assuming that a distance threshold (such as 5 pixels) is set; then, all detection frames with a distance less than the distance threshold are determined, and all detection frames satisfying the condition are taken as the detection frame matching the motion information of the target object.

[0111] In some embodiments, the position of the target object is predicted based on the motion information of the target object, and the predicted position of the target object in the current frame image is obtained, which can be achieved by the following method: obtaining the motion information of the target object, wherein the motion information comprises at least one of the horizontal pixel position and the horizontal direction velocity of the center of the tracking box associated with the target object, the vertical pixel position and the vertical direction velocity of the center of the tracking box associated with the target object, the area and the area change velocity of the tracking box associated with the target object, and the aspect ratio of the tracking box associated with the target object; constructing a state model based on the motion information; and calling the state model to predict the position of the target object to obtain the predicted position of the target object in the current frame image.

[0112] It should be noted that the motion information of the target object can be determined according to the first M frame images before the current frame image (wherein M is a positive integer greater than 1) and the change of the position information of the target object in the first M frame images; the motion information of the target object can also be directly obtained according to a preset interface; the specific way of obtaining the motion information of the target object can be determined according to actual conditions, which is not limited in the embodiments of the present application.

[0113] In some embodiments, first, key features such as velocity change rate, acceleration change rate, direction change, etc. are extracted from the obtained motion data; then, a filter such as Kalman filter, particle filter, etc. is used to estimate the motion state of the target object; then, a mathematical model is established to describe the motion behavior of the target object according to the extracted features and the state estimation results; finally, the accuracy and reliability of the state model are verified by comparing with the actual data. The state model of the target object can be established by formula (1).

[0114] wherein u represents the horizontal pixel position of the center of the tracking box associated with the target object, v represents the vertical pixel position of the center of the tracking box associated with the target object, s represents the area of the tracking box associated with the target object, and r represents the aspect ratio of the tracking box associated with the target object, represents the horizontal direction velocity of the center of the tracking box associated with the target object, represents the vertical direction velocity of the center of the tracking box associated with the target object, represents the area change velocity of the tracking box associated with the target object.

[0115] In some embodiments, the position of the target object in the current frame image can be predicted by using the state model in combination with the current motion information of the target object. Here, the prediction of the position can involve integrating the velocity or other more complex prediction algorithms to obtain the predicted position of the target object in the current frame image.

[0116] In some embodiments, the distance between each detection box and the predicted position can be determined by the following method: for each detection box, subtract the position of the detection box from the predicted position to obtain a second subtraction result; transpose the second subtraction result to obtain a transposed second subtraction result; obtain a covariance matrix between the position of the detection box and the predicted position; multiply the transposed second subtraction result and the inverse matrix of the covariance matrix to obtain a second multiplication result; multiply the second multiplication result and the second subtraction result, and take the multiplication result as the distance between the detection box and the predicted position.

[0117] For example, taking the jthdetection box in the current frame image as an example, assuming that the position corresponding to the detection box is d j , and the predicted position of the target object is y, subtracting the position of the detection box from the predicted position of the target object obtains a second subtraction result (d j -y); transposing the second subtraction result (d j -y) obtains a transposed second subtraction result (d j -y) T ; obtaining a covariance matrix S between the position d j of the detection box and the predicted position y of the target object; multiplying the transposed second subtraction result (d j -y) T and the inverse matrix S -1 of the covariance matrix obtains a second multiplication result (d j -y) T S -1 ; multiplying the second multiplication result (d j -y) T S -1 and the second subtraction result (d j -y) obtains the distance (d j -y) T S -1 between the detection box and the predicted position (d j -y).

[0118] In some embodiments, the feature queue may store N appearance features corresponding to the target object in the first N frames of images, where N is a positive integer greater than 1. The determination of the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue can also be achieved in the following way: transpose the feature vector corresponding to the appearance features of the object to obtain the transposed feature vector; multiply the transposed feature vector by the feature vectors corresponding to the N appearance features respectively to obtain N first multiplication results; subtract the preset value from the N first multiplication results respectively to obtain N first subtraction results; and take the minimum value among the N first subtraction results as the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue.

[0119] For example, taking the j-th detection box in the current frame image as an example, assume that the feature vector corresponding to the appearance features of the object in the j-th detection box is r. j The feature vector r corresponding to the appearance features of the object j Transpose the vector to obtain the transposed eigenvector r. j T The transposed eigenvector r j T The feature vector r corresponding to the kth appearance feature k Performing the multiplication, we obtain the result of the kth first multiplication as r. j T r k For each of the N appearance features, calculate the product of the features with the transposed feature vector to obtain N first product results; assuming a preset value of 1, multiply the preset value with the kth first product result to obtain the corresponding kth first subtraction result (1-r). j T r k For each of the N first multiplication results, subtract the preset value from each of the N first subtraction results to obtain N first subtraction results; then find the minimum value among the N subtraction results, i.e., min{1-r}. j T r k |r k ∈R}, and use it as the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue.

[0120] In some embodiments, the step 1041A described above can also be implemented by: determining, from the at least one detection frame, a detection frame matching the motion information of the target object; when the number of the matching detection frames is multiple, performing feature extraction on the objects respectively included in the multiple matching detection frames to obtain appearance features respectively corresponding to the objects; respectively determining similarities between the appearance features of each object and the appearance features of the target object pre-stored in the feature queue; and when the number of the similarities greater than the second similarity threshold is one, taking the detection frame corresponding to the similarity greater than the second similarity threshold as the target detection frame including the target object.

[0121] It should be noted that the implementation manner of the above when the number of the matching detection frames is multiple in the embodiments of the present application is similar to the implementation manner when the number of the matching detection frames is single, and details can be referred to the implementation manner when the number of the matching detection frames is single, which will not be described herein again.

[0122] For example, assuming that the number of the matching detection frames is 3 (for example, assuming that the detection frame 1, the detection frame 2 and the detection frame 3), the similarities between the appearance features of the objects respectively included in each detection frame and the appearance features of the target object pre-stored in the feature queue are 0.6, 0.7 and 0.8 respectively, and the second similarity threshold is 0.75, then the number of the similarities greater than the second similarity threshold is one (i.e. the detection frame 3), that is, the detection frame (i.e. the detection frame 3) corresponding to the similarity 0.8 is taken as the target detection frame including the target object.

[0123] In some embodiments, when the number of the similarities greater than the second similarity threshold is multiple, and the number of the similarities greater than a third similarity threshold in the multiple similarities greater than the second similarity threshold is one, the detection frame corresponding to the similarity greater than the third similarity threshold is taken as the target detection frame including the target object.

[0124] Here, the third similarity threshold is greater than the second similarity threshold.

[0125] For example, assuming that the number of the matching detection frames is 3 (for example, assuming that the detection frame 1, the detection frame 2 and the detection frame 3), the similarities between the appearance features of the objects respectively included in each detection frame and the appearance features of the target object pre-stored in the feature queue are 0.6, 0.8 and 0.9 respectively, the second similarity threshold is 0.75, and the third similarity threshold is 0.85, then the number of the similarities greater than the second similarity threshold is two (i.e. the detection frame 2 and the detection frame 3), and the number of the similarities greater than the third similarity threshold in the multiple similarities greater than the second similarity threshold is one (i.e. the detection frame 3), then the detection frame (i.e. the detection frame 3) corresponding to the similarity 0.9 is taken as the target detection frame including the target object.

[0126] In step 1041B, when the detection state represents that the target object is lost in the current frame image, a target detection frame including the target object is determined from the at least one detection frame based on the appearance feature of the target object pre-stored.

[0127] In some embodiments, when the target object is lost in the current frame image, a target detection frame including the target object can be determined from the detection frame of the current frame image according to the appearance feature of the target object.

[0128] In some embodiments, referring to FIG. 6, FIG. 6 is a flowchart of a target detection method provided in an embodiment of the present application, as shown in FIG. 6, the step 1041B shown in FIG. 5 can be implemented by steps 10411B to 10414B shown in FIG. 6, which will be described in combination with the steps shown in FIG. 6.

[0129] In step 10411B, the objects included in the at least one detection frame are respectively subjected to feature extraction, to obtain the appearance feature corresponding to each object.

[0130] It should be noted that the implementation of step 10411B in the embodiment of the present application is similar to the implementation of the object appearance feature extraction described above, and specific reference can be made to the implementation of the object appearance feature extraction described above, which will not be repeated here.

[0131] In step 10412B, the similarity between the appearance feature of each object and the appearance feature of the target object pre-stored is respectively determined.

[0132] It should be noted that the implementation of step 10412B in the embodiment of the present application is similar to the implementation of the calculation of the similarity between the appearance features described above, and specific reference can be made to the implementation of the calculation of the similarity between the appearance features described above, which will not be repeated here.

[0133] In step 10413B, when the number of similarities greater than the fourth similarity threshold value is one, the detection frame corresponding to the similarity greater than the fourth similarity threshold value is taken as the target detection frame including the target object.

[0134] Here, the fourth similarity threshold value can be the same as or different from the second similarity threshold value described above, and can be determined according to actual conditions.

[0135] It should be noted that the implementation of step 10413B in the embodiments of the present application is similar to the implementation of the above-mentioned case where the number of similarities greater than the second similarity threshold is one, and the bounding box corresponding to the similarity greater than the second similarity threshold is taken as the target bounding box including the target object, and details can be referred to the implementation of the above-mentioned case where the number of similarities greater than the second similarity threshold is one, and the bounding box corresponding to the similarity greater than the second similarity threshold is taken as the target bounding box including the target object, which will not be repeated here.

[0136] In step 10414B, when the number of similarities greater than the fourth similarity threshold is multiple, and the number of similarities greater than the fifth similarity threshold in the multiple similarities greater than the fourth similarity threshold is one, the bounding box corresponding to the similarity greater than the fifth similarity threshold is taken as the target bounding box including the target object.

[0137] Here, the fifth similarity threshold is greater than the fourth similarity threshold, and the fifth similarity threshold can be the same as the third similarity threshold described above or different, which can be determined according to actual conditions.

[0138] It should be noted that the implementation of step 10414B in the embodiments of the present application is similar to the implementation of the above-mentioned case where the number of similarities greater than the second similarity threshold is multiple, and the number of similarities greater than the third similarity threshold in the multiple similarities greater than the second similarity threshold is one, and the bounding box corresponding to the similarity greater than the third similarity threshold is taken as the target bounding box including the target object, and details can be referred to the implementation of the above-mentioned case where the number of similarities greater than the second similarity threshold is multiple, and the number of similarities greater than the third similarity threshold in the multiple similarities greater than the second similarity threshold is one, and the bounding box corresponding to the similarity greater than the third similarity threshold is taken as the target bounding box including the target object, which will not be repeated here.

[0139] In some embodiments, the appearance features of the target object are stored in a pre-created feature queue, see FIG. 7, which is a flowchart of a target detection method provided in an embodiment of the present application. As shown in FIG. 7, after step 104 shown in FIG. 3 is executed, steps 105 to 107 shown in FIG. 7 can also be executed, which will be described in combination with the steps shown in FIG. 7.

[0140] In step 105, the target object included in the target bounding box is feature-extracted to obtain the latest appearance features of the target object.

[0141] In some embodiments, the target object included in the target bounding box can be feature-extracted to obtain the latest appearance features corresponding to the target object included in the target bounding box, i.e., the appearance features of the target object in the current image frame.

[0142] In step 106, when the number of appearance features of the target object stored in the feature queue is less than the number threshold, the latest appearance feature is inserted at the tail of the feature queue.

[0143] In some embodiments, when the number of appearance features of the target object stored in the feature queue is less than the number threshold, it indicates that there is still storage space in the current feature queue, and thus the latest appearance feature can be directly inserted at the tail of the feature queue.

[0144] For example, assuming that the latest appearance feature A corresponding to the target detection frame is obtained, and the maximum storage capacity of the feature queue is 100, when the number of appearance features stored in the feature queue is less than 100, it indicates that there is still storage space in the current feature queue, and thus the appearance feature A is directly inserted at the tail of the feature queue.

[0145] In step 107, when the number of appearance features of the target object stored in the feature queue is equal to the number threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted at the tail of the feature queue.

[0146] In some embodiments, when the number of appearance features of the target object stored in the feature queue is equal to the number threshold, it indicates that the storage space in the current feature queue is full, and thus the appearance feature at the head of the feature queue needs to be deleted, and the latest appearance feature is inserted at the tail of the feature queue to realize the storage of the appearance feature.

[0147] For example, assuming that the latest appearance feature B corresponding to the target detection frame is obtained, and the maximum storage capacity of the feature queue is 100, when the number of appearance features stored in the feature queue is equal to 100, it indicates that the storage space in the current feature queue is full, and thus the appearance feature C at the head of the feature queue is deleted, and the latest appearance feature B is inserted at the tail of the feature queue.

[0148] In some embodiments, in addition to using the feature queue to store the appearance features of the target object, other data structure types such as stacks, linked lists, arrays, etc. can also be used to store the appearance features of the target object, which are not specifically limited here.

[0149] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described. The exemplary application describes the specific implementation process of the target detection method in a single target tracking scenario.

[0150] To solve the problem of slow processing speed in target tracking according to motion information and appearance information, the application selectively determines whether to use appearance feature extraction for tracking by judging the tracking state of the target object, thereby combining the classic multi-target tracking algorithm using appearance feature extraction for tracking and the multi-target tracking algorithm not using appearance feature extraction for tracking, and effectively balancing the detection speed and detection effect of the algorithm.

[0151] In the target object tracking process, the application uses a matching method based on motion information to realize target exclusive tracking, and a feature queue is established for the target object to save the feature information (such as appearance features) of the target object in the last 100 frames. When a frame of image is obtained each time, it is judged whether the target object is lost. When the target object is determined to be not lost, the tracking of most objects in the picture is completed through motion information. After the target object is matched through motion information, a Re-identification (ReID) model is used to extract the features of the target object included in the possible detection box, and the extracted appearance features are compared with the appearance features of the target object saved in the feature queue. When the similarity meets the similarity threshold, it is confirmed that the object included in the detection box is the target object, and the latest extracted feature is saved to the feature queue. Otherwise, it is confirmed that the target object does not exist in the current frame image. When the target object is not found for a plurality of consecutive frames, it is determined that the target object is lost, and a ReID model is used to extract the features of all objects in the picture. The extracted features are compared with the appearance features of the target object saved in the feature queue. When there is a certain appearance feature whose similarity with the feature queue meets the similarity threshold, it is confirmed that the target object is found back and the target object is the object corresponding to the appearance feature.

[0152] The following takes the Simple Online and Realtime Tracking (SORT) algorithm as an example to introduce the target detection method in the single target tracking scene in modules.

[0153] In the motion estimation unit, the SORT algorithm uses a linear constant velocity model independent of other objects and camera motion to approximate the inter-frame displacement of each object, and constructs a state model according to the motion information of the target object. The modeling can be realized according to the above formula (1). The position of the target object in the current frame can be predicted by the constructed state model to realize the prediction of the future position of the target object.

[0154] In the appearance feature updating unit, the SORT algorithm establishes a feature queue for the tracker corresponding to the tracked target, which is used to save the appearance feature information of the target object corresponding to the tracker in the last 100 frames. The kth feature information in the tracker is denoted as r k In the current image frame, when the tracker is associated with a detection box, the appearance feature r j of the object included in the detection box is updated into the feature queue. When the available capacity of the feature queue is less than 100, r j is directly inserted into the tail of the feature queue; otherwise, r0 in the head of the feature queue is deleted, and r j is inserted into the tail of the feature queue.

[0155] In the data association unit, when the detection box is assigned to an existing tracker, the motion information and the appearance information are used to match the target detection box.

[0156] (1) The motion association degree D 0 (i,j) between the detection box and the target object prediction box can be described by calculating the Mahalanobis distance between the detection box and the target object prediction box by formula (2). 0 (i,j)=(d j -y) T S -1 (d j -y) (2)

[0157] where d j j represents the position of the jth detection box in the current frame image, y represents the position of the target object prediction box, and S is the covariance matrix between the position d j of the detection box and the position y of the target object prediction box.

[0158] (2) For the target object, the ReID model is used to extract and save the appearance feature. The appearance feature verification is performed on the result obtained by the SORT algorithm every frame. When the target object is lost for a certain time length, the appearance feature of all objects in the picture is extracted, and the saved appearance feature is used to find the target. The minimum cosine distance D 1 (trackid,j) between the feature vector corresponding to the appearance feature of the object in the detection box and the feature vector corresponding to the appearance feature of the target object can also be calculated by formula (3).

[0159] where r j is the feature vector corresponding to the appearance feature of the object in the jth detection box, r k is the feature vector corresponding to the kth appearance feature in the feature queue, and trackid represents the identity code of the target object.

[0160] (3) When there is only one detection box j in the current frame image whose Mahalanobis distance to the target object prediction box meets the minimum Mahalanobis distance threshold, it is determined whether the appearance similarity of the detection box j to the target object meets the loose appearance distance constraint (corresponding to the first similarity threshold described above): if it meets, the object included in the detection box j is determined as the target object; if it does not meet, it is determined that the target object is lost in the current frame image.

[0161] When there are multiple detection boxes in the current frame image whose Mahalanobis distance to the target object prediction box meets the minimum Mahalanobis distance threshold, it is further determined how many detection boxes meet the medium appearance distance constraint: if only one detection box p meets the minimum Mahalanobis distance threshold and the similarity meets the medium appearance distance constraint (corresponding to the second similarity threshold described above), the object included in the detection box p is determined as the target object; otherwise, the strict appearance distance constraint (corresponding to the third similarity threshold described above) is used to determine the target detection box for all detection boxes in the current frame image, and when the similarity of the detector q to the target object prediction box meets the strict appearance distance constraint, the object included in the detection box p is determined as the target object, otherwise, it is determined that the target object is lost in the current frame image.

[0162] (4) The target object prediction box is matched with the detection box by using the cascade matching and the minimum cost algorithm.

[0163] In the creating and deleting tracker unit, when the object enters and leaves the image, the tracker needs to be created or destroyed accordingly, and when any detection box with an overlap with any existing tracking box is less than the minimum IOU threshold, it indicates that there is an untracked object. A new tracker is created for the untracked object. At the same time, the new tracker needs to undergo a probation period of n frames, and the tracker needs to be associated with the detection box to accumulate enough evidence to prevent false tracking; if there is a tracker that has not been successfully associated with the detection box for x consecutive frames, the tracking of the tracker is terminated to prevent the unlimited growth of the number of trackers and the positioning error due to the long prediction time without detection correction, wherein when the tracking target of the tracker is not the target object, x can be set to be smaller (for example, it can be set to 5), and when the tracking target of the tracker is the target object, x can be set to be larger to ensure that when the tracking target disappears from the picture due to improper operation (occlusion or walking out of the camera area) and then returns to the picture, the tracking target can be found again.

[0164] In the searching state switching unit, the algorithm utilizes the motion information and the appearance information to realize the association of the detection frame and the tracker, the motion information can make up for the deficiency of the appearance information, which is helpful to the stability of the matching; when the followed target disappears for a long time, the credibility of the estimation of the motion information of the algorithm is reduced, even if the followed target reappears in the picture, the followed target cannot be found back, thus it is necessary to switch the searching state; when the followed target continuously appears in the picture, the algorithm belongs to the normal state, and the motion information and the appearance information are simultaneously utilized to realize the association of the detection frame and the tracker; when the followed target disappears for a long time in the picture, the algorithm enters the searching state, and only the appearance information is utilized to realize the association of the detection frame and the tracker.

[0165] The following continues to describe an exemplary structure of the target detection device 555 provided by the embodiments of the present application, which is implemented as a software module. In some embodiments, as shown in FIG. 2, the software module stored in the target detection device 555 of the memory 550 can include an acquisition module 5551, an identification module 5552, and a determination module 5553.

[0166] The acquisition module 5551 is configured to acquire a current frame image, where the current frame image is a video frame image acquired at a current time; the identification module 5552 is configured to identify at least one detection frame from the current frame image, where each detection frame includes an object; and the determination module 5553 is configured to determine a detection state of a target object in the current frame image, where the detection state represents whether the target object is lost in the current frame image, and the target object is a to-be-detected object in the at least one object; and the determination module 5553 is further configured to determine a target detection frame including the target object from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object according to the detection state.

[0167] In some embodiments, the determination module 5553 is further configured to determine the target detection frame including the target object from the at least one detection frame based on the motion information of the target object and the pre-stored appearance features of the target object when the detection state represents that the target object is not lost in the current frame image.

[0168] In some embodiments, the determination module 5553 is further configured to determine a detection frame matching the motion information of the target object from the at least one detection frame; when the number of the matching detection frames is one, perform feature extraction on an object included in the matching detection frame to obtain appearance features of the object; determine a similarity between the appearance features of the object and pre-stored appearance features of the target object in a feature queue; and when the similarity is greater than a first similarity threshold, determine the matching detection frame as the target detection frame including the target object.

[0169] In some embodiments, the feature queue stores N appearance features of the target object corresponding to N previous frames of images respectively, where N is a positive integer greater than 1, and the determining module 5553 is further configured to transpose a feature vector corresponding to the appearance feature of the object to obtain a transposed feature vector; multiply the transposed feature vector with the feature vectors corresponding to the N appearance features respectively to obtain N first multiplication results respectively; subtract a preset value from the N first multiplication results respectively to obtain N first subtraction results respectively; and take the minimum value in the N first subtraction results as a similarity between the appearance feature of the object and the appearance features of the target object pre-stored in the feature queue.

[0170] In some embodiments, the determining module 5553 is further configured to perform position prediction on the target object based on the motion information of the target object to obtain a predicted position of the target object in the current frame of image; determine distances between the at least one detection frame and the predicted position respectively; and take the detection frame with a distance less than a distance threshold as the detection frame matching the motion information of the target object.

[0171] In some embodiments, the determining module 5553 is further configured to obtain the motion information of the target object, where the motion information includes at least one of: a horizontal pixel position and a horizontal direction velocity of a tracking frame center associated with the target object, a vertical pixel position and a vertical direction velocity of the tracking frame center associated with the target object, an area and an area change velocity of the tracking frame associated with the target object, and an aspect ratio of the tracking frame associated with the target object; construct a state model based on the motion information; and call the state model to perform position prediction on the target object to obtain a predicted position of the target object in the current frame of image.

[0172] In some embodiments, the determining module 5553 is further configured to subtract the position of the detection frame from the predicted position to obtain a second subtraction result; transpose the second subtraction result to obtain a transposed second subtraction result; obtain a covariance matrix between the position of the detection frame and the predicted position; multiply the transposed second subtraction result with an inverse matrix of the covariance matrix to obtain a second multiplication result; multiply the second multiplication result with the second result, and take the obtained multiplication result as the distance between the detection frame and the predicted position.

[0173] In some embodiments, the determining module 5553 is further configured to determine the detection frame matching the motion information of the target object from the at least one detection frame; when the number of the matching detection frames is greater than one, perform feature extraction on objects included in the plurality of matching detection frames to obtain appearance features corresponding to the objects respectively; determine similarities between the appearance features of each object and the appearance features of the target object pre-stored in the feature queue respectively; and when the number of the similarities greater than a second similarity threshold is one, take the detection frame corresponding to the similarity greater than the second similarity threshold as the target detection frame including the target object.

[0174] In some embodiments, the determining module 5553 is further configured to, when the number of similarities greater than the second similarity threshold is multiple, and the number of similarities greater than a third similarity threshold among the multiple similarities greater than the second similarity threshold is one, determine the detection frame corresponding to the similarity greater than the third similarity threshold as the target detection frame including the target object, where the third similarity threshold is greater than the second similarity threshold.

[0175] In some embodiments, the determining module 5553 is further configured to, when the detection state indicates that the target object is lost in the current frame image, determine the target detection frame including the target object from the at least one detection frame based on the pre-stored appearance feature of the target object.

[0176] In some embodiments, the determining module 5553 is further configured to perform feature extraction on the objects included in the at least one detection frame respectively to obtain appearance features respectively corresponding to the at least one object; determine similarities between the appearance feature of each object and the pre-stored appearance feature of the target object respectively; when the number of similarities greater than a fourth similarity threshold is one, determine the detection frame corresponding to the similarity greater than the fourth similarity threshold as the target detection frame including the target object; when the number of similarities greater than the fourth similarity threshold is multiple, and the number of similarities greater than a fifth similarity threshold among the multiple similarities greater than the fourth similarity threshold is one, determine the detection frame corresponding to the similarity greater than the fifth similarity threshold as the target detection frame including the target object, where the fifth similarity threshold is greater than the fourth similarity threshold.

[0177] In some embodiments, the appearance feature of the target object is stored in a pre-created feature queue, and the target detection device further includes a storage module 5554 configured to perform feature extraction on the target object included in the target detection frame to obtain a latest appearance feature of the target object; when the number of appearance features of the target object stored in the feature queue is less than a quantity threshold, insert the latest appearance feature at the tail of the feature queue; and when the number of appearance features of the target object stored in the feature queue is equal to the quantity threshold, delete the appearance feature at the head of the feature queue and insert the latest appearance feature at the tail of the feature queue.

[0178] In some embodiments, the determining module 5553 is further configured to obtain an association state between a tracker associated with the target object and the detection frame; when the association state indicates that the tracker has an associated detection frame, or when the association state indicates that the time length for which the tracker does not have an associated detection frame is less than a time length threshold, determine that the target object is not lost in the current frame image; and when the association state indicates that the time length for which the tracker does not have an associated detection frame is greater than the time length threshold, determine that the target object is lost in the current frame image.

[0179] It should be noted that the description of the device of the embodiments of the present application is similar to the description of the above-mentioned method embodiments, has similar beneficial effects as the method embodiments, and therefore will not be described again. For technical details not described in the target detection device provided by the embodiments of the present application, it can be understood according to the description of any one of FIG. 3, FIG. 4, FIG. 5, FIG. 6, or FIG. 7.

[0180] The embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the target detection method provided by the embodiments of the present application.

[0181] The embodiments of the present application provide a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions are stored in the computer readable storage medium. When the computer executable instructions are executed by the processor, the processor will execute the target detection method provided by the embodiments of the present application, for example, the target detection method shown in FIG. 3, FIG. 4, FIG. 5, FIG. 6, or FIG. 7.

[0182] In some embodiments, the computer readable storage medium can be a ferroelectric memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM) and the like. It can also be various devices including one or any combination of the above memories.

[0183] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0184] As an example, the executable instructions can be deployed in one electronic device, or in multiple electronic devices located at one site, or in multiple electronic devices distributed over multiple sites and interconnected through a communication network.

[0185] In summary, the embodiments of the present application have the following beneficial effects:

[0186] (1) The problem of poor tracking effect when target objects cross in the scheme of using only motion information for multi-target tracking to implement a single-target tracking algorithm is improved;

[0187] (2) The problem that the target cannot be found when the target object disappears from the screen in the scheme of using only motion information for multi-target tracking to implement a single-target tracking algorithm is solved;

[0188] (3) The problem of slow algorithm speed caused by the need to extract features from all objects in the screen in the scheme of using motion information and appearance information for multi-target tracking to implement a single-target tracking algorithm is solved.

[0189] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A target detection method characterized by, The method comprises: acquiring a current frame image, wherein the current frame image is a video frame image collected at a current time; identifying at least one detection box from the current frame image, wherein each detection box comprises an object; determining a detection state of a target object in the current frame image, wherein the detection state represents whether the target object is lost in the current frame image, and the target object is an object to be detected among the at least one object; determining a target detection box comprising the target object from the at least one detection box according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object.

2. The method of claim 1, wherein, The determining a target detection box comprising the target object from the at least one detection box according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object, comprises: when the detection state represents that the target object is not lost in the current frame image, determining a target detection box comprising the target object from the at least one detection box based on the motion information of the target object and the pre-stored appearance features of the target object.

3. The method of claim 2, wherein, The determining a target detection box comprising the target object from the at least one detection box based on the motion information of the target object and the pre-stored appearance features of the target object comprises: determining a detection box matching the motion information of the target object from the at least one detection box; when the number of the matching detection boxes is one, performing feature extraction on an object comprised in the matching detection box to obtain appearance features of the object; determining a similarity between the appearance features of the object and pre-stored appearance features of the target object in a feature queue; when the similarity is greater than a first similarity threshold, determining the matching detection box as the target detection box comprising the target object.

4. The method of claim 3, wherein: the feature queue stores N appearance features of the target object corresponding to N previous frame images respectively, wherein N is a positive integer greater than 1; the determining a similarity between the appearance features of the object and pre-stored appearance features of the target object in a feature queue comprises: transposing a feature vector corresponding to the appearance features of the object to obtain a transposed feature vector; multiplying the transposed feature vector with feature vectors corresponding to the N appearance features respectively to obtain N first multiplication results correspondingly; subtracting a preset value from the N first multiplication results respectively to obtain N first subtraction results correspondingly; taking a minimum value in the N first subtraction results as the similarity between the appearance features of the object and the pre-stored appearance features of the target object in the feature queue.

5. The method of claim 3, wherein, The determining a detection box matching the motion information of the target object from the at least one detection box comprises: predicting a position of the target object based on the motion information of the target object, to obtain a predicted position of the target object in the current frame image; respectively determining distances between the at least one detection frame and the predicted position; taking the detection frame with a distance less than a distance threshold as a detection frame matching the motion information of the target object.

6. The method of claim 5, wherein, The predicting a position of the target object based on the motion information of the target object, to obtain a predicted position of the target object in the current frame image, comprises: obtaining the motion information of the target object, wherein the motion information comprises at least one of: horizontal pixel position and horizontal direction velocity of a tracking frame center associated with the target object, vertical pixel position and vertical direction velocity of the tracking frame center associated with the target object, area and area change velocity of the tracking frame associated with the target object, aspect ratio of the tracking frame associated with the target object; constructing a state model based on the motion information; calling the state model to predict a position of the target object, to obtain a predicted position of the target object in the current frame image.

7. The method of claim 5, wherein, The respectively determining distances between the at least one detection frame and the predicted position comprises: performing the following processing for each detection frame: subtracting the position of the detection frame from the predicted position to obtain a second subtraction result; transposing the second subtraction result to obtain a transposed second subtraction result; obtaining a covariance matrix between the position of the detection frame and the predicted position; multiplying the transposed second subtraction result and an inverse matrix of the covariance matrix to obtain a second multiplication result; multiplying the second multiplication result and the second subtraction result, and taking the obtained multiplication result as the distance between the detection frame and the predicted position.

8. The method of claim 2, wherein, The determining a target detection frame including the target object from the at least one detection frame based on the motion information of the target object and the pre-stored appearance feature of the target object comprises: determining a detection frame matching the motion information of the target object from the at least one detection frame; when the number of the matching detection frames is multiple, performing feature extraction on objects respectively included in the multiple matching detection frames to obtain appearance features respectively corresponding to the objects; respectively determining similarities between the appearance feature of each object and the pre-stored appearance feature of the target object in a feature queue; when the number of the similarities greater than a second similarity threshold is one, taking the detection frame corresponding to the similarity greater than the second similarity threshold as the target detection frame including the target object.

9. The method of claim 8, wherein, The method further comprises: when the number of the similarities greater than the second similarity threshold is multiple, and the number of the similarities greater than a third similarity threshold in the multiple similarities greater than the second similarity threshold is one, taking the detection frame corresponding to the similarity greater than the third similarity threshold as the target detection frame including the target object, wherein the third similarity threshold is greater than the second similarity threshold.

10. The method according to any one of claims 1 to 9, characterized in that, The target detection frame including the target object is determined from the at least one detection frame based on the motion information of the target object and the pre-stored appearance feature of the target object, or based on the pre-stored appearance feature of the target object according to the detection state, and the method comprises the following steps. When the detection state indicates that the target object is lost in the current frame image, the target detection frame including the target object is determined from the at least one detection frame based on the pre-stored appearance feature of the target object.

11. The method of claim 10, wherein, The target detection frame including the target object is determined from the at least one detection frame based on the pre-stored appearance feature of the target object, and the method comprises the following steps. Feature extraction is performed on the objects included in the at least one detection frame respectively to obtain at least one appearance feature corresponding to each of the objects respectively; The similarity between the appearance feature of each of the objects and the pre-stored appearance feature of the target object is determined respectively; When the number of the similarities greater than the fourth similarity threshold value is one, the detection frame corresponding to the similarity greater than the fourth similarity threshold value is taken as the target detection frame including the target object; When the number of the similarities greater than the fourth similarity threshold value is more than one, and the number of the similarities greater than a fifth similarity threshold value is one in the similarities greater than the fourth similarity threshold value, the detection frame corresponding to the similarity greater than the fifth similarity threshold value is taken as the target detection frame including the target object, wherein the fifth similarity threshold value is greater than the fourth similarity threshold value.

12. The method of any one of claims 1 to 9, wherein: The appearance feature of the target object is stored in a pre-created feature queue; After the target detection frame including the target object is determined from the at least one detection frame, the method further comprises: Feature extraction is performed on the target object included in the target detection frame to obtain a latest appearance feature of the target object; When the number of the appearance features of the target object stored in the feature queue is less than a number threshold value, the latest appearance feature is inserted into the tail of the feature queue; When the number of the appearance features of the target object stored in the feature queue is equal to the number threshold value, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted into the tail of the feature queue.

13. The method of any one of claims 1 to 9, wherein: The detection state of the target object in the current frame image is determined, and the method comprises the following steps. An association state between a tracker associated with the target object and a detection frame is obtained; When the association state indicates that the tracker has an associated detection frame, or when the time length for which the tracker does not have an associated detection frame is less than a time length threshold value, it is determined that the target object is not lost in the current frame image; When the time length for which the tracker does not have an associated detection frame is greater than the time length threshold value, it is determined that the target object is lost in the current frame image.

14. A target detection apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire a current frame image, wherein the current frame image is a video frame image collected at a current time point; An identification module is configured to identify at least one detection box from the current frame image, wherein each detection box includes an object; A determination module is configured to determine a detection state of a target object in the current frame image, wherein the detection state represents whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object; The determination module is further configured to determine a target detection box including the target object from the at least one detection box according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object.

15. An electronic device, comprising: comprise: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the target detection method in any one of claims 1 to 13.

16. A computer-readable storage medium storing computer-executable instructions, wherein execution of the computer-executable instructions by one or more processors of a computing system causes the one or more processors to perform operations comprising: The computer executable instructions are executed by the processor to implement the target detection method in any one of claims 1 to 13.

Images