Target detection method and device, electronic equipment, computer readable storage medium and computer program product
By combining motion information and appearance features on a low-computing-power end-side robot, a target detection method is developed to identify and manage detection boxes, solving the balance problem between detection speed and effectiveness, and achieving stable target tracking.
Patent Information
- Application Number
- CN202411020993.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-29
AI Technical Summary
Existing technologies cannot achieve a balance between detection speed and detection performance in two-dimensional multi-target tracking algorithms on low-computing-power edge robots, leading to the easy loss of targets.
By acquiring the current frame image, the detection box is identified and the detection state of the target object is determined. Combining the motion information of the target object with the pre-stored appearance features, the target detection box is determined from the detection box. The appearance features are stored and managed using a feature queue, and the cascaded matching strategy is used to optimize computing resources.
While maintaining detection speed, the accuracy and stability of target detection are improved, target loss is avoided, and the algorithm achieves a balance between detection speed and performance.
Smart Images

Figure CN119048554A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target detection method, device, electronic device, computer-readable storage medium, and computer program product. Background Art
[0002] The two-dimensional multi-target tracking algorithm is divided into a method based on motion information matching and a method based on motion information and appearance information matching according to whether appearance information is used. Among them, the method based on motion information matching cannot find the target after the human body walks out of the picture (that is, the detection effect is poor), and when the method based on motion information and appearance information is deployed to a low-computing-power end robot, the overall calculation speed of the algorithm will slow down, which will make it easy to lose the target. In other words, the relevant technology cannot achieve a balance between the detection speed and the detection effect of the algorithm. Summary of the invention
[0003] The embodiments of the present application provide a target detection method, device, electronic device, computer-readable storage medium and computer program product, which can effectively strike a balance between the detection speed and detection effect of the algorithm.
[0004] The technical solution of the embodiment of the present application is implemented as follows:
[0005] The present application provides a target detection method, including:
[0006] Acquire a current frame image, wherein the current frame image is a video frame image acquired at the current moment;
[0007] Identify at least one detection frame from the current frame image, wherein each of the detection frames includes an object;
[0008] Determining a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected among the at least one object;
[0009] According to the detection state, based on the motion information of the target object and the pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object, a target detection frame including the target object is determined from the at least one detection frame.
[0010] The present application provides a target detection device, including:
[0011] An acquisition module is used to acquire a current frame image, wherein the current frame image is a video frame image acquired at a current moment;
[0012] A recognition module, configured to recognize at least one detection frame from the current frame image, wherein each of the detection frames includes an object;
[0013] a determination module, configured to determine a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object;
[0014] The determination module is also used to determine a target detection frame including the target object from the at least one detection frame according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on pre-stored appearance features of the target object.
[0015] In the above scheme, the determination module is also used to determine a target detection frame including the target object from the at least one detection frame based on the motion information of the target object and the pre-stored appearance features of the target object when the detection state represents that the target object is not lost in the current frame image.
[0016] In the above scheme, the determination module is also used to determine a detection frame that matches the motion information of the target object from the at least one detection frame; when the number of the matched detection frames is one, perform feature extraction on the object included in the matched detection frame to obtain the appearance features of the object; determine the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue; when the similarity is greater than a first similarity threshold, determine the matched detection frame as a target detection frame including the target object.
[0017] In the above scheme, the feature queue stores N appearance features corresponding to the target object in the previous N frames of images, wherein N is a positive integer greater than 1, and the determination module is further used to transpose the feature vector corresponding to the appearance feature of the object to obtain the transposed feature vector; multiply the transposed feature vector with the feature vectors corresponding to the N appearance features to obtain N first multiplication results; subtract the preset values from the N first multiplication results to obtain N first subtraction results; and use the minimum value of the N first subtraction results as the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue.
[0018] In the above scheme, the determination module is also used to predict the position of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image; respectively determine the distance between the at least one detection frame and the predicted position; and use the detection frame whose distance is less than the distance threshold as the detection frame that matches the motion information of the target object.
[0019] In the above scheme, the determination module is also used to obtain motion information of the target object, wherein the motion information includes at least one of the following: the horizontal pixel position and horizontal speed of the center of the tracking frame associated with the target object, the vertical pixel position and vertical speed of the center of the tracking frame associated with the target object, the area and area change speed of the tracking frame associated with the target object, and the aspect ratio of the tracking frame associated with the target object; constructing a state model based on the motion information; calling the state model to predict the position of the target object to obtain the predicted position of the target object in the current frame image.
[0020] In the above scheme, the determination module is also used to subtract the position of the detection frame from the predicted position to obtain a second subtraction result; transpose the second subtraction result to obtain a transposed second subtraction result; obtain the covariance matrix between the position of the detection frame and the predicted position; multiply the transposed second subtraction result with the inverse matrix of the covariance matrix to obtain a second multiplication result; multiply the second multiplication result with the second subtraction result, and use the obtained multiplication result as the distance between the detection frame and the predicted position.
[0021] In the above scheme, the determination module is also used to determine a detection frame that matches the motion information of the target object from the at least one detection frame; when the number of the matched detection frames is multiple, feature extraction is performed on the objects respectively included in the multiple matched detection frames to obtain appearance features corresponding to the multiple objects; the similarity between the appearance features of each object and the appearance features of the target object pre-stored in the feature queue is determined respectively; when the number of similarities greater than a second similarity threshold is one, the detection frame corresponding to the similarity greater than the second similarity threshold is used as a target detection frame including the target object.
[0022] In the above scheme, the determination module is also used to use the detection frame corresponding to the similarity greater than the third similarity threshold as the target detection frame including the target object when the number of the similarities greater than the second similarity threshold is multiple and among the multiple similarities greater than the second similarity threshold, the number of the similarities greater than the third similarity threshold is one, wherein the third similarity threshold is greater than the second similarity threshold.
[0023] In the above scheme, the determination module is also used to determine a target detection frame including the target object from the at least one detection frame based on pre-stored appearance features of the target object when the detection state indicates that the target object is lost in the current frame image.
[0024] In the above scheme, the determination module is also used to extract features of the objects respectively included in the at least one detection frame to obtain appearance features corresponding to at least one of the objects; respectively determine the similarity between the appearance features of each of the objects and the appearance features of the target object stored in advance; when the number of similarities greater than a fourth similarity threshold is one, use the detection frame corresponding to the similarity greater than the fourth similarity threshold as a target detection frame including the target object; when the number of similarities greater than the fourth similarity threshold is multiple, and among the multiple similarities greater than the fourth similarity threshold, the number of similarities greater than a fifth similarity threshold is one, use the detection frame corresponding to the similarity greater than the fifth similarity threshold as a target detection frame including the target object, wherein the fifth similarity threshold is greater than the fourth similarity threshold.
[0025] In the above scheme, the appearance features of the target object are stored in a pre-created feature queue, and the device also includes a storage module for extracting features of the target object included in the target detection frame to obtain the latest appearance features of the target object; when the number of appearance features of the target object stored in the feature queue is less than a quantity threshold, the latest appearance feature is inserted to the end of the feature queue; when the number of appearance features of the target object stored in the feature queue is equal to the quantity threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted to the end of the feature queue.
[0026] In the above scheme, the determination module is also used to obtain the association status between the tracker and the detection frame associated with the target object; when the association status represents that the tracker has an associated detection frame, or when the association status represents that the duration for which the tracker does not have an associated detection frame is less than a duration threshold, it is determined that the target object is not lost in the current frame image; when the association status represents that the duration for which the tracker does not have an associated detection frame is greater than the duration threshold, it is determined that the target object is lost in the current frame image.
[0027] An embodiment of the present application provides an electronic device, including:
[0028] A memory for storing executable instructions;
[0029] The processor is used to implement a target detection method provided in an embodiment of the present application when executing the executable instructions stored in the memory.
[0030] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing a target detection method provided by an embodiment of the present application when executed by a processor.
[0031] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement a target detection method provided by an embodiment of the present application when executed by a processor.
[0032] The embodiments of the present application have the following beneficial effects:
[0033] By acquiring the current frame image, identifying at least one detection frame from the current frame image, and determining the detection status of the target object in the current frame image, wherein the detection status characterizes whether the target object is lost in the current frame image, and finally determining a target detection frame including the target object from at least one detection frame based on the detection status, based on a combination of motion information and appearance features, or based solely on appearance features. That is to say, the embodiments of the present application use different methods to detect the target object by judging whether the target object is lost in the current frame image, thereby being able to effectively strike a balance between the detection speed and detection effect of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 is a schematic diagram of the architecture of the target detection system 100 provided in an embodiment of the present application;
[0035] Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;
[0036] Figure 3 It is a flowchart of the target detection method provided in the embodiment of the present application;
[0037] Figure 4 It is a flowchart of the target detection method provided in the embodiment of the present application;
[0038] Figure 5 It is a flowchart of the target detection method provided in the embodiment of the present application;
[0039] Figure 6 It is a flowchart of the target detection method provided in the embodiment of the present application;
[0040] Figure 7 It is a flowchart of the target detection method provided in the embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0043] It is understandable that in the embodiments of the present application, when user information and other related data (such as user characteristic information) are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.
[0044] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0045] In the following description, the terms "first\second\..." involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0047] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0048] 1) Tracker: A tracker is an algorithm or software module used to continuously identify and track one or more targets in a video stream or other sensor data.
[0049] 2) Tracking box: In the target tracking algorithm, the tracking box is a tool used to define the area of the object to be tracked in the image. The tracking box is usually a rectangle, which detects the position of the object to be tracked in the video or image data.
[0050] 3) Detection box: In the target tracking algorithm, the tracking box is a tool used to define the area of the object in the image. The detection box is usually a rectangle, which usually detects the position of all objects in the video or image data to match the detected object with the object to be tracked.
[0051] 4) Feature queue: A feature queue is a data structure used to process image or video data in computer vision. It is essentially a queue that can store feature data extracted from images or video frames. The feature queue is stored according to the "first in, first out" storage principle, that is, the feature data to be stored is inserted from the end of the feature queue for storage. When deleting, the feature data that entered the feature queue earliest will be deleted from the head of the feature queue.
[0052] 5) Intersection over Union (IOU) threshold: The IOU threshold is an important parameter used to evaluate the overlap between two bounding boxes in target detection and tracking. In target tracking tasks, the IOU threshold is used to evaluate the match between the target position predicted by the tracker and the true target position. When the IOU between the predicted target position and the true target position is greater than the set threshold, the two positions are considered to match, that is, the predicted target position is correct.
[0053] 6) Cascade matching: Cascade matching is a strategy that is usually used in scenarios that require efficient processing of large-scale data or rapid iteration of model prediction results. Its core idea is to design a series of simple and fast screening steps (or models) to filter out unlikely matching results at different stages, leaving only the best matching results. Through an orderly hierarchical arrangement, the number of candidate matches that ultimately require detailed evaluation is greatly reduced, thereby saving overall computing resource consumption and improving processing efficiency.
[0054] Target tracking algorithms are divided into two categories: single target tracking algorithms and multi-target tracking algorithms. Among them, the single target tracking algorithm only focuses on the tracking target itself during the tracking process, without combining the information of other human bodies in the picture. Therefore, the tracking effect is poor when human bodies cross, and long-term tracking cannot be achieved (that is, the target cannot be found after walking out of the picture); while the multi-target tracking algorithm pays attention to the appearance information of all human bodies in the picture during the tracking process. When human bodies cross, the motion information of multiple human bodies can be combined to obtain the optimal match, which can effectively improve the problem of poor tracking effect when human bodies cross, and further combine the appearance information of human bodies for tracking and matching, which can further improve the tracking effect when human bodies cross, and can also achieve target recovery after the target human body walks out of the picture.
[0055] Two-dimensional multi-target tracking algorithms are mainly divided into two methods according to whether appearance information is used: matching based on motion information and matching based on motion information and appearance information.
[0056] When using a motion information matching-based tracking method, representative algorithms include SORT and ByteTrack. It is necessary to use a target detection algorithm to detect the human body in each frame. Although they combine the motion information of multiple human bodies to obtain the best match and improve the performance when the human body crosses, when the trajectory of the human body is close, there is a detection error, or there is a posture change, it is still possible that an ID switch will occur when crossing, and there is no way to retrieve the target after the human body walks out of the picture.
[0057] When using the classic tracking method based on motion information and appearance information matching, the representative algorithm is Deep SORT. In addition to detecting the human body in each frame, it is also necessary to extract features of all human bodies in the picture. When matching, it is also necessary to determine whether the motion information is continuous (whether the motion information meets the threshold) and whether the appearance is similar (similarity meets the threshold). This can further improve the tracking errors caused by factors such as detection errors and posture changes during the human intersection process. When the target human body walks out of the picture, the stored appearance information can be combined to retrieve the target. However, this algorithm requires one human body detection and multiple human feature extractions in each frame. When the algorithm runs on a low-computing-power end-side robot, the overall calculation speed of the algorithm becomes very slow, which makes it easy to lose the target.
[0058] Based on this, the embodiments of the present application provide a target detection method, device, electronic device, computer-readable storage medium and computer program product, which can effectively strike a balance between the detection speed and detection effect of the algorithm. The electronic device provided in the embodiments of the present application can be implemented as a server, or implemented by a server and a terminal in collaboration. The following is an example of the target detection method provided by the embodiment of the present application being implemented in collaboration with a server and a terminal.
[0059] For example, see Figure 1 , Figure 1 is a schematic diagram of the architecture of the target detection system 100 provided in the embodiment of the present application, in order to realize supporting a target detection application, such as Figure 1 As shown, the target detection system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, wherein the network 300 can be a local area network or a wide area network, or a combination of the two; the terminal 400 is a terminal associated with a user, and a client 410 runs on the terminal 400. The client 410 can be various types of clients, such as a dedicated target detection client, a robot navigation client, etc.
[0060] In some embodiments, the user can collect the video frame image at the current moment through the client 410, and the terminal 400 transmits the acquired video frame image to the server 200 through the network 300. Then, the server 200 identifies at least one detection frame from the current frame image, wherein each detection frame includes an object; then, the server 200 determines the detection state of the target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected in at least one object; then, the server 200 can determine the target detection frame including the target object from at least one detection frame according to the detection state, based on the motion information of the target object and the appearance features of the target object stored in advance, or based on the appearance features of the target object stored in advance, so as to detect the target object. Finally, the server 200 sends the target detection frame including the target object to the terminal 400 through the network 300, and presents it on the client 410.
[0061] Of course, the above process can also be implemented by the terminal 400 alone. For example, the terminal 400 can determine the target detection frame including the target object from at least one detection frame based on its own computing power. The embodiment of the present application does not make any specific limitations on this.
[0062] It should be noted that the technical solution provided by this application can be applied to various scenarios, such as autonomous driving, human-computer interaction, robot navigation, etc. For example, applying the target detection scenario provided by this application to the robot target tracking scenario with limited end-side computing power can achieve stable tracking of specific objects; in games and virtual reality applications, it can achieve tracking of user gestures or body movements to provide a more natural and intuitive user interaction experience; in the scenario of autonomous driving vehicles, it can achieve identification and tracking of other vehicles, pedestrians or any potential obstacles on the road, and by accurately tracking these targets, it can better determine the vehicle's driving path and speed to ensure driving safety. The embodiments of this application do not specifically limit the application scenarios of the target detection method.
[0063] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.
[0064] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.
[0065] For example, Figure 1 The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a robot, etc., but is not limited thereto. The terminal 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0066] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a terminal as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .
[0067] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0068] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0069] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.
[0070] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.
[0071] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0072] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0073] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB), etc.;
[0074] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);
[0075] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.
[0076] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The target detection device 555 stored in the memory 550 is shown, which can be software in the form of a program or plug-in, including the following software modules: an acquisition module 5551, an identification module 5552, a determination module 5553 and a storage module 5554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be noted that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the target detection device 555 excludes the implementation that can only include the acquisition module 5551, the recognition module 5552 and the determination module 5553. The functions of each module will be explained below.
[0077] The target detection method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the terminal device provided in the embodiment of the present application.
[0078] See also Figure 3 , Figure 3 is a flow chart of the target detection method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.
[0079] It should be noted that Figure 3 The method shown can be executed by various forms of computer programs running on the terminal device, and is not limited to the client. For example, it can also be the operating system, software module, script, and applet described above, so the example of the client in the following should not be regarded as a limitation on the embodiments of the present application. In addition, for the sake of convenience, the following does not specifically distinguish between the terminal and the client running on the terminal.
[0080] In step 101, a current frame image is obtained.
[0081] Here, the current frame image may refer to a video frame image acquired at the current moment.
[0082] In some embodiments, the image at the current moment can be captured in real time by a camera, or the video frame image at the current moment can be captured in real time by a camera or other video capture device.
[0083] For example, if you want to obtain the current video frame image in a real-time video stream, you can use the cv2.VideoCapture(0) method to obtain the video stream from the camera. It should be noted that in actual applications, you need to ensure that you have permission to access the required video file or camera.
[0084] In step 102, at least one detection frame is identified from the current frame image.
[0085] Here, each detection box may include an object.
[0086] In some embodiments, first, the current video frame image is converted into a format suitable for a deep learning model (for example, the video frame image size can be adjusted, normalized, converted to floating point numbers, etc.), and image enhancement technology (such as cropping, rotation, scaling, etc.) can also be used to improve the detection effect and realize preprocessing of the video frame image; then, a deep learning framework (such as TensorFlow, PyTorch, OpenCV, etc.) is used to load the trained target detection model to ensure that the model has completed training and can run normally on the given input data; then, the preprocessed video frame image is input into the target detection model for forward propagation calculation, and the model outputs the corresponding prediction results (such as target category, confidence, and target bounding box coordinates, etc.); then, the bounding box coordinates, category and confidence are parsed for the prediction results of the model. Since a single object may be detected by multiple bounding boxes, it is necessary to eliminate overlapping bounding boxes until the bounding box that is most likely to contain the target object is retained; finally, a corresponding detection box is drawn according to the bounding box on the video frame image, and the drawn detection box is displayed for subsequent target detection.
[0087] In step 103, the detection status of the target object in the current frame image is determined.
[0088] Here, the detection state can be used to indicate whether the target object is lost in the current frame image, and the target object is an object to be detected in at least one object (eg, user 1).
[0089] In some embodiments, see Figure 4 , Figure 4is a flow chart of the target detection method provided in the embodiment of the present application, such as Figure 4 As shown, Figure 3 Step 103 shown may be performed by Figure 4 Steps 1031 to 1033 shown in the figure are implemented by combining Figure 4 The steps shown are explained.
[0090] In step 1031, the association relationship between the tracker associated with the target object and the detection box is obtained.
[0091] Here, the association relationship can be used to characterize whether the tracker associated with the target object (eg, user 1) has an associated detection box.
[0092] In some embodiments, during the target tracking process, the tracker interacts with the detection frame to determine and identify the target object to be tracked. When the tracker has an associated detection frame, it means that the tracker has a detection frame matching the target object associated with the tracker in the current frame image; when the tracker does not have an associated detection frame, it means that the tracker does not have a detection frame matching the target object associated with the tracker in the current frame image.
[0093] In step 1032, when the associated state representation tracker has an associated detection frame, or when the duration for which the associated state representation tracker does not have an associated detection frame is less than a duration threshold, it is determined that the target object is not lost in the current frame image.
[0094] Here, when the target object is not lost in the current frame image or the loss duration is short, it is considered that the target object is not lost in the current frame image.
[0095] In some embodiments, when the associated state represents that there is an associated detection frame for the tracker, it means that the target object is detected in the current frame image and is not lost.
[0096] In some embodiments, when the duration of the associated state representation tracker without an associated detection frame is less than a duration threshold, it means that although the target object has not been detected in the current frame image, the target object's loss duration is short and does not reach the duration threshold, and it is considered that the target object has not been lost in the current frame image.
[0097] In step 1033, when the duration of the detection frame for which the associated state represents the absence of the associated tracker is greater than the duration threshold, it is determined that the target object is lost in the current frame image.
[0098] In some embodiments, when the duration of the detection frame without the associated state representation tracker being greater than the duration threshold, it means that the target object has disappeared for a long time, and it is considered that the target object has been lost in the current frame image.
[0099] Here, when the target object is lost in the current frame image for a long time, it is considered that the target object has been lost in the current frame image.
[0100] In step 104, according to the detection state, based on the motion information of the target object and the pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object, a target detection frame including the target object is determined from at least one detection frame.
[0101] Here, the detection state includes two states: the target object is not lost in the current frame image and the target object is lost in the current frame image.
[0102] In some embodiments, see Figure 5 , Figure 5 is a flow chart of the target detection method provided in the embodiment of the present application, such as Figure 5 As shown, Figure 3 Step 104 shown may be performed by Figure 5 Step 1041A or step 1041B shown in the figure is implemented by combining Figure 5 The steps shown are explained.
[0103] In step 1041A, when the detection state indicates that the target object is not lost in the current frame image, a target detection frame including the target object is determined from at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object.
[0104] In some embodiments, when the target object is not lost in the current frame image, a target detection frame including the target object can be determined from at least one detection frame included in the current frame image based on motion information and appearance features of the target object.
[0105] In some embodiments, the above-mentioned step 1041A can be implemented in the following manner: determine a detection frame that matches the motion information of the target object from at least one detection frame; when the number of matched detection frames is one, perform feature extraction on the object included in the matched detection frame to obtain the appearance features of the object; determine the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue; when the similarity is greater than a first similarity threshold, determine the matched detection frame as a target detection frame including the target object.
[0106] For example, assuming that the first similarity threshold is 0.5, and assuming that the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue is 0.7, that is, greater than the first similarity threshold 0.5, the matched detection frame can be determined as a target detection frame including the target object.
[0107] In some embodiments, when the number of matched detection frames is one, the object included in the matched detection frame is directly feature extracted to obtain the appearance features of the object (such as color, texture, shape, edge information, etc.); in addition, when the appearance feature dimension of the object is high, representative features can be selected, or the appearance feature dimension can be reduced through dimensionality reduction methods (such as principal component analysis, etc.).
[0108] In some embodiments, the above-mentioned determination of a detection frame that matches the motion information of the target object from at least one detection frame can be achieved in the following way: predicting the position of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image; determining the distance between at least one detection frame and the predicted position respectively; and using the detection frame whose distance is less than a distance threshold as the detection frame that matches the motion information of the target object.
[0109] For example, assume that a distance threshold (eg, 5 pixels) is set; then, all detection frames whose distance is less than the distance threshold are determined, and all detection frames that meet the condition are used as detection frames that match the motion information of the target object.
[0110] In some embodiments, the above-mentioned position prediction of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image can be achieved in the following way: obtaining the motion information of the target object, wherein the motion information includes at least one of the following: the horizontal pixel position and horizontal speed of the center of the tracking frame associated with the target object, the vertical pixel position and vertical speed of the center of the tracking frame associated with the target object, the area and area change speed of the tracking frame associated with the target object, and the aspect ratio of the tracking frame associated with the target object; constructing a state model based on the motion information; calling the state model to predict the position of the target object to obtain the predicted position of the target object in the current frame image.
[0111] It should be noted that the operation information of the target object can be determined based on the previous M frame images (where M is a positive integer greater than 1) before the current frame image, and based on the change in the position information of the target object in the previous M frame images; the motion information of the target object can also be directly obtained according to a preset interface; the specific method of obtaining the motion information of the target object can be determined according to the actual situation, and the embodiments of the present application are not specifically limited here.
[0112] In some embodiments, key features (such as velocity change rate, acceleration change rate, direction change, etc.) are first extracted from the acquired motion data; then, a filter (such as a Kalman filter, a particle filter, etc.) is used to estimate the motion state of the target object; then, based on the extracted features and the state estimation results, a mathematical model is established to describe the motion behavior of the target object; finally, the accuracy and reliability of the state model are verified by comparing with the actual data. The state model of the target object can be established by formula (1).
[0113]
[0114] Among them, u represents the horizontal pixel position of the center of the tracking frame associated with the target object, v represents the vertical pixel position of the center of the tracking frame associated with the target object, s represents the area of the tracking frame associated with the target object, and r represents the aspect ratio of the tracking frame associated with the target object. Indicates the horizontal speed of the center of the tracking box associated with the target object. Indicates the vertical velocity of the center of the tracking box associated with the target object. Indicates how fast the area of the tracking box associated with the target object changes.
[0115] In some embodiments, the state model can be used in combination with the current motion information of the target object to predict the position of the target object in the current frame image. This requires integrating the velocity or other more complex prediction algorithms to predict the position and obtain the predicted position of the target object in the current frame image.
[0116] In some embodiments, the above-mentioned determination of the distance between at least one detection frame and the predicted position can be achieved in the following manner: performing the following processing for each detection frame: subtracting the position of the detection frame from the predicted position to obtain a second subtraction result; transposing the second subtraction result to obtain a transposed second subtraction result; obtaining the covariance matrix between the position of the detection frame and the predicted position; multiplying the transposed second subtraction result with the inverse matrix of the covariance matrix to obtain a second multiplication result; multiplying the second multiplication result with the second subtraction result, and using the obtained multiplication result as the distance between the detection frame and the predicted position.
[0117] For example, take the jth detection box in the current frame image as an example, assuming that the position corresponding to the detection box is d j , the predicted position of the target object is y, the position of the detection box is subtracted from the predicted position of the target object to obtain the second subtraction result (d j -y); the second subtraction result (d j -y) is transposed, and the second subtraction result after transposition is (d j -y)T ; Get the position of the detection box d j The covariance matrix S between the predicted position y of the target object; the transposed second subtraction result (d j -y) T and the inverse matrix S of the covariance matrix -1 Multiply them to get the second multiplication result (d j -y) T S -1 ; The second multiplication result (d j -y) T S -1 Subtract the result from the second j -y) to obtain the distance (d) between the detection box and the predicted position j -y) T S -1 (d j -y).
[0118] In some embodiments, the feature queue may store N appearance features corresponding to the target object in the previous N frames of images, where N is a positive integer greater than 1. The above-mentioned determination of the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue may also be achieved in the following manner: transposing the feature vector corresponding to the appearance feature of the object to obtain the transposed feature vector; multiplying the transposed feature vector with the feature vectors corresponding to the N appearance features, respectively, to obtain N first multiplication results; subtracting the preset values from the N first multiplication results, respectively, to obtain N first subtraction results; and taking the minimum value of the N first subtraction results as the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue.
[0119] For example, taking the jth detection frame in the current frame image as an example, assuming that the feature vector corresponding to the appearance feature of the object in the jth detection frame is r j , the feature vector r corresponding to the appearance feature of the object j Transpose and get the transposed eigenvector r j T ; Transpose the eigenvector r j T The feature vector r corresponding to the kth appearance feature k Multiply them and get the kth first multiplication result as r j T r k ; Calculate the multiplication results between the N appearance features and the transposed feature vector respectively, and obtain N first multiplication results; Assuming that the preset value is 1, multiply the preset value with the kth first multiplication result, and obtain the kth first subtraction result (1-rj T r k ); for the N first multiplication results, the preset value is subtracted from the N first subtraction results to obtain N first subtraction results; the minimum value among the N subtraction results, i.e., min{1-r j T r k |r k ∈R}, and takes it as the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue.
[0120] In some embodiments, the above-mentioned step 1041A can also be implemented in the following manner: determine a detection frame that matches the motion information of the target object from at least one detection frame; when the number of matched detection frames is multiple, perform feature extraction on the objects respectively included in the multiple matching detection frames to obtain appearance features corresponding to the multiple objects; determine the similarity between the appearance features of each object and the appearance features of the target object pre-stored in the feature queue; when the number of similarities greater than the second similarity threshold is one, use the detection frame corresponding to the similarity greater than the second similarity threshold as the target detection frame including the target object.
[0121] It should be noted that the above-mentioned implementation method when the number of matched detection frames is multiple in the embodiment of the present application is similar to the implementation method when the number of matched detection frames is single. For details, please refer to the implementation method when the number of matched detection frames is single, and the embodiment of the present application will not be repeated here.
[0122] For example, assuming that the number of matched detection frames is 3 (for example, detection frame 1, detection frame 2 and detection frame 3), the similarities between the appearance features of the object included in each detection frame and the appearance features of the target object pre-stored in the feature queue are 0.6, 0.7, and 0.8, respectively, and assuming that the second similarity threshold is 0.75, the number of similarities greater than the second similarity threshold is one (i.e., detection frame 3), that is, the detection frame corresponding to the similarity of 0.8 (i.e., detection frame 3) is used as the target detection frame including the target object.
[0123] In some embodiments, when there are multiple similarities greater than the second similarity threshold, and among the multiple similarities greater than the second similarity threshold, the number of similarities greater than the third similarity threshold is one, the detection frame corresponding to the similarity greater than the third similarity threshold is used as the target detection frame including the target object.
[0124] Here, the third similarity threshold is greater than the second similarity threshold.
[0125] For example, assuming that the number of matched detection frames is 3 (for example, detection frame 1, detection frame 2 and detection frame 3), the similarities between the appearance features of the object included in each detection frame and the appearance features of the target object pre-stored in the feature queue are 0.6, 0.8, and 0.9, respectively, assuming that the second similarity threshold is 0.75 and the third similarity threshold is 0.85, then the number of similarities greater than the second similarity threshold is two (i.e., detection frame 2 and detection frame 3), and among the multiple similarities greater than the second similarity threshold, the number of similarities greater than the third similarity threshold is one (i.e., detection frame 3), then the detection frame corresponding to the similarity of 0.9 (i.e., detection frame 3) is used as the target detection frame including the target object.
[0126] In step 1041B, when the detection status indicates that the target object is lost in the current frame image, a target detection frame including the target object is determined from at least one detection frame based on pre-stored appearance features of the target object.
[0127] In some embodiments, when the target object is lost in the current frame image, a target detection frame including the target object may be determined from the detection frame of the current frame image according to appearance features of the target object.
[0128] In some embodiments, see Figure 6 , Figure 6 is a flow chart of the target detection method provided in the embodiment of the present application, such as Figure 6 As shown, Figure 5 Step 1041B shown may be performed by Figure 6 Steps 10411B to 10414B shown in the figure are implemented by combining Figure 6 The steps shown are explained.
[0129] In step 10411B, feature extraction is performed on objects respectively included in at least one detection frame to obtain appearance features corresponding to at least one object.
[0130] It should be noted that the implementation of step 10411B in the embodiment of the present application is similar to the implementation of the above-mentioned object appearance feature extraction. For details, please refer to the above-mentioned implementation of the object appearance feature extraction, which will not be repeated here.
[0131] In step 10412B, the similarity between the appearance features of each object and the pre-stored appearance features of the target object is determined respectively.
[0132] It should be noted that the implementation of step 10412B in the embodiment of the present application is similar to the implementation of calculating the similarity between the appearance features described above. For details, please refer to the implementation of calculating the similarity between the appearance features described above, which will not be repeated here.
[0133] In step 10413B, when the number of similarities greater than the fourth similarity threshold is one, the detection frame corresponding to the similarity greater than the fourth similarity threshold is used as the target detection frame including the target object.
[0134] Here, the fourth similarity threshold may be the same as or different from the second similarity threshold, and may be determined according to actual conditions.
[0135] It should be noted that the implementation method of step 10413B in the embodiment of the present application is similar to the above-mentioned implementation method of when the number of similarities greater than the second similarity threshold is one, the detection box corresponding to the similarity greater than the second similarity threshold is used as the target detection box including the target object. Specifically, please refer to the above-mentioned implementation method of when the number of similarities greater than the second similarity threshold is one, the detection box corresponding to the similarity greater than the second similarity threshold is used as the target detection box including the target object, which will not be repeated here.
[0136] In step 10414B, when there are multiple similarities greater than the fourth similarity threshold, and among the multiple similarities greater than the fourth similarity threshold, the number of similarities greater than the fifth similarity threshold is one, the detection frame corresponding to the similarity greater than the fifth similarity threshold is used as the target detection frame including the target object.
[0137] Here, the fifth similarity threshold is greater than the fourth similarity threshold. The fifth similarity threshold may be the same as or different from the third similarity threshold, and may be determined according to actual conditions.
[0138] It should be noted that the implementation method of step 10414B in the embodiment of the present application is similar to the above-mentioned implementation method of using the detection box corresponding to the similarity greater than the third similarity threshold as the target detection box including the target object when the number of similarities greater than the second similarity threshold is multiple and the number of similarities greater than the third similarity threshold among the multiple similarities greater than the second similarity threshold is one. Specifically, please refer to the above-mentioned implementation method of using the detection box corresponding to the similarity greater than the third similarity threshold as the target detection box including the target object when the number of similarities greater than the second similarity threshold is multiple and the number of similarities greater than the third similarity threshold among the multiple similarities greater than the second similarity threshold is one, which will not be repeated here.
[0139] In some embodiments, the appearance features of the target object are stored in a pre-created feature queue, see Figure 7 , Figure 7 is a flow chart of the target detection method provided in the embodiment of the present application, such as Figure 7 As shown, in the execution Figure 3After step 104 shown, you can also perform Figure 7 Steps 105 to 107 shown in FIG. 106 will be combined with Figure 7 The steps shown are explained.
[0140] In step 105, feature extraction is performed on the target object included in the target detection frame to obtain the latest appearance features of the target object.
[0141] In some embodiments, feature extraction may be performed on the target object included in the target detection frame to obtain the latest appearance features corresponding to the target object included in the target detection frame, that is, the appearance features of the target object in the current image frame.
[0142] In step 106 , when the number of appearance features of the target object stored in the feature queue is less than the number threshold, the latest appearance feature is inserted into the end of the feature queue.
[0143] In some embodiments, when the number of appearance features of the target object stored in the feature queue is less than the number threshold, it means that there is still storage space in the current feature queue, and the latest appearance feature can be directly inserted into the end of the feature queue.
[0144] For example, assuming that the latest appearance feature A corresponding to the target detection frame is obtained, the maximum storage capacity of the feature queue is 100. When the number of appearance features stored in the feature queue is less than 100, it means that the current feature queue still has storage space, then the appearance feature A is directly inserted into the end of the feature queue.
[0145] In step 107 , when the number of appearance features of the target object stored in the feature queue is equal to the number threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted into the tail of the feature queue.
[0146] In some embodiments, when the number of appearance features of the target object stored in the feature queue is equal to a quantity threshold, it means that the storage space in the current feature queue is full, and it is necessary to delete the appearance feature at the head of the feature queue and insert the latest appearance feature to the end of the feature queue to achieve storage of the appearance features.
[0147] For example, suppose that the latest appearance feature B corresponding to the target detection frame is obtained, and the maximum storage capacity of the feature queue is 100. When the number of stored appearance features in the feature queue is equal to 100, it means that the current feature queue storage space is full, then the appearance feature C at the head of the feature queue is deleted, and the latest appearance feature B is inserted to the end of the feature queue.
[0148] In some embodiments, in addition to using a feature queue to store the appearance features of the target object, other data structure types such as a stack, a linked list, an array, etc. may also be used to store the appearance features of the target object, which is not specifically limited here.
[0149] Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described. The exemplary application describes the specific implementation process of the target detection method in a single target tracking scenario.
[0150] In order to solve the problem of slow processing speed in target tracking based on motion information and appearance information, the present application judges the tracking state of the target object and selectively determines whether to use appearance feature extraction for tracking, thereby combining the classic multi-target tracking algorithm that uses appearance feature extraction for tracking with the multi-target tracking algorithm that does not use appearance feature extraction for tracking, effectively achieving a balance between the algorithm's detection speed and detection effect.
[0151] In the target object tracking process, the present application uses a matching method based on motion information as a whole to achieve exclusive target tracking, and establishes a separate feature queue for the target object to save the feature information (such as appearance features) of the target object in the most recent 100 frames. Each time a frame of image is obtained, it is determined whether the target object is lost. When the target object is determined to be not lost, the motion information is first used to complete the tracking of most objects in the picture. After the possible detection frame of the target object is obtained through motion information matching, a re-identification (Re-identification, ReID) model is used to extract features of the target object included in the possible detection frame, and the extracted appearance features are calculated for similarity with the appearance features of the target object stored in the feature queue. When the similarity meets the similarity threshold, it is confirmed that the object included in the detection frame is the target object, and the latest extracted features are saved in the feature queue; otherwise, it is confirmed that the target object does not exist in the current frame image. When the target object is not found in multiple consecutive frames, it is determined that the target object is lost, and the ReID model is used to extract features of all objects in the picture. The similarity of all extracted features is calculated with the appearance features of the target object stored in the feature queue. When there is an appearance feature whose similarity with the feature queue meets the similarity threshold, it is confirmed that the target object is found and the target object is the object corresponding to the appearance feature.
[0152] The following takes the Simple Online and Realtime Tracking (SORT) algorithm as an example to specifically introduce the target detection method in the single target tracking scenario by module.
[0153] In the motion estimation unit, the SORT algorithm uses a linear constant velocity model that is independent of other objects and camera motion to approximate the inter-frame displacement of each object, and constructs a state model based on the motion information of the target object. Specifically, the modeling can be implemented according to the above formula (1). The constructed state model can be used to predict the position of the target object in the current frame to achieve the prediction of the future position of the target object.
[0154] In the appearance feature update unit, the SORT algorithm establishes a feature queue for the tracker corresponding to the tracked target, which is used to save the appearance feature information of the target object corresponding to the tracker in the latest 100 frames. The kth feature information in the tracker is represented as r k , in the current image frame, when the tracker is associated with a detection box, the appearance feature r of the object included in the detection box is j Update to the feature queue. When the available capacity of the feature queue is less than 100, r j Insert it to the end of the feature queue; otherwise, you need to insert the r at the head of the feature queue 0 Delete and then r j Insert to the end of the feature queue.
[0155] In the data association unit, motion information and appearance information are used to match the target detection box when assigning the detection box to the existing tracker.
[0156] (1) The Mahalanobis distance between the detection box and the target object prediction box can be calculated by formula (2) to describe the degree of motion correlation between the detection box and the target object prediction box. 0 (i,j).
[0157] D 0 (o,j)=(d j -y) T S -1 (d j -y) (2)
[0158] Among them, d j represents the position corresponding to the jth detection box in the current frame image, y represents the position of the target object prediction box, and S is the position of the detection box d j The covariance matrix between the position y of the predicted box of the target object.
[0159] (2) For the target object, use the ReID model to extract and save the appearance features. The appearance features of the results obtained by the SORT algorithm are verified for each frame. When the target object is lost for a certain period of time, the appearance features of all objects in the picture are extracted and the target is found using the saved appearance features. The minimum cosine distance D between the feature vector corresponding to the appearance features of the object in the detection frame and the feature vector corresponding to the appearance features of the target object can also be calculated using formula (3):1 (trackid,j).
[0160]
[0161] Among them, r j is the feature vector corresponding to the appearance feature of the object in the jth detection frame, r k is the feature vector corresponding to the kth appearance feature in the feature queue, and trackid represents the identity identification code corresponding to the target object.
[0162] (3) When only one detection box j in the current frame image has a Mahalanobis distance with the target object prediction box that meets the minimum Mahalanobis distance threshold, it is determined whether the appearance feature similarity between the detection box j and the target object meets the loose appearance distance constraint (corresponding to the first similarity threshold mentioned above): if so, it is determined that the object included in the detection box j is the target object; if not, it is determined that the target object in the current frame image is lost.
[0163] When the Mahalanobis distances between multiple detection frames and the target object prediction frame in the current frame image satisfy the minimum Mahalanobis distance threshold, it is further determined how many detection frames and the target object prediction frame have Mahalanobis distances that satisfy the medium appearance distance constraint: if only one detection frame p and the target object prediction frame have a Mahalanobis distance that satisfies the minimum Mahalanobis distance threshold and the similarity satisfies the medium appearance distance constraint (corresponding to the second similarity threshold mentioned above), the object included in the detection frame p is determined as the target object; otherwise, a strict appearance distance constraint (corresponding to the third similarity threshold mentioned above) is used for all detection frames in all current frame images to determine the target detection frame, and when the similarity between the detector q and the target object prediction frame satisfies the strict appearance distance constraint, the object included in the detection frame p is determined as the target object, otherwise, it is determined that the target object in the current frame image is lost.
[0164] (4) Use cascade matching and minimum cost algorithm to match the target object prediction box with the detection box.
[0165] In the tracker creation and deletion unit, when an object enters or leaves the image, a tracker needs to be created or destroyed accordingly. When any detection frame that overlaps with any existing tracking frame is less than the minimum IOU threshold, it indicates that there is an untracked object. A new tracker is created for the untracked object. At the same time, the new tracker needs to go through a trial period of n frames. The tracker needs to be associated with the detection frame to accumulate sufficient evidence to prevent false positive tracking. If there is a tracker that has not been successfully associated with the detection frame for x consecutive frames, the tracking of the tracker is terminated to prevent the unlimited growth of the number of trackers and the positioning error caused by the long prediction time without detection correction. When the tracking target of the tracker is not the target object, the setting of x can be relatively small (for example, it can be set to 5). When the tracking target of the tracker is the target object, the setting of x can be relatively large to ensure that the tracking target can be found again when it returns to the picture after disappearing from the picture due to improper operation (occlusion or walking out of the camera area).
[0166] In the search state switching unit, the algorithm uses both motion information and appearance information to associate the detection frame and the tracker. The motion information can make up for the lack of appearance information, which helps to stabilize the matching. When the follow target disappears for too long, the algorithm's estimated credibility of its motion information decreases. Even if the follow target reappears in the picture, the follow target cannot be found, so the search state needs to be switched. When the follow target appears continuously in the picture, the algorithm is in a normal state, and uses both motion information and appearance information to associate the detection frame and the tracker. When the follow target disappears in the picture for a period of time greater than a certain length, the algorithm enters a search state, and only uses appearance information to associate the detection frame and the tracker.
[0167] The following further describes an exemplary structure of the target detection device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the target detection device 555 of the memory 550 may include: an acquisition module 5551 , an identification module 5552 , and a determination module 5553 .
[0168] An acquisition module 5551 is used to acquire a current frame image, wherein the current frame image is a video frame image acquired at a current moment; an identification module 5552 is used to identify at least one detection frame from the current frame image, wherein each detection frame includes an object; a determination module 5553 is used to determine a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected in at least one object; the determination module 5553 is further used to determine a target detection frame including a target object from at least one detection frame according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on pre-stored appearance features of the target object.
[0169] In some embodiments, the determination module 5553 is also used to determine a target detection frame including the target object from at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object when the detection state indicates that the target object is not lost in the current frame image.
[0170] In some embodiments, the determination module 5553 is also used to determine a detection frame that matches the motion information of the target object from at least one detection frame; when the number of matched detection frames is one, perform feature extraction on the object included in the matched detection frame to obtain the appearance features of the object; determine the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue; when the similarity is greater than a first similarity threshold, determine the matched detection frame as a target detection frame that includes the target object.
[0171] In some embodiments, the feature queue stores N appearance features corresponding to the target object in the previous N frames of images, where N is a positive integer greater than 1. The determination module 5553 is also used to transpose the feature vector corresponding to the appearance feature of the object to obtain the transposed feature vector; multiply the transposed feature vector with the feature vectors corresponding to the N appearance features, and obtain N first multiplication results; subtract the preset values from the N first multiplication results, and obtain N first subtraction results; and use the minimum value of the N first subtraction results as the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue.
[0172] In some embodiments, the determination module 5553 is also used to predict the position of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image; respectively determine the distance between at least one detection frame and the predicted position; and use the detection frame whose distance is less than the distance threshold as the detection frame that matches the motion information of the target object.
[0173] In some embodiments, the determination module 5553 is also used to obtain motion information of the target object, wherein the motion information includes at least one of the following: the horizontal pixel position and horizontal speed of the center of the tracking frame associated with the target object, the vertical pixel position and vertical speed of the center of the tracking frame associated with the target object, the area and area change speed of the tracking frame associated with the target object, and the aspect ratio of the tracking frame associated with the target object; construct a state model based on the motion information; call the state model to predict the position of the target object to obtain the predicted position of the target object in the current frame image.
[0174] In some embodiments, the determination module 5553 is also used to subtract the position of the detection frame from the predicted position to obtain a second subtraction result; transpose the second subtraction result to obtain a transposed second subtraction result; obtain the covariance matrix between the position of the detection frame and the predicted position; multiply the transposed second subtraction result with the inverse matrix of the covariance matrix to obtain a second multiplication result; multiply the second multiplication result with the second result, and use the obtained multiplication result as the distance between the detection frame and the predicted position.
[0175] In some embodiments, the determination module 5553 is also used to determine a detection frame that matches the motion information of the target object from at least one detection frame; when the number of matched detection frames is multiple, feature extraction is performed on the objects respectively included in the multiple matching detection frames to obtain appearance features corresponding to the multiple objects; the similarity between the appearance features of each object and the appearance features of the target object pre-stored in the feature queue is determined respectively; when the number of similarities greater than the second similarity threshold is one, the detection frame corresponding to the similarity greater than the second similarity threshold is used as the target detection frame including the target object.
[0176] In some embodiments, the determination module 5553 is also used to, when there are multiple similarities greater than the second similarity threshold, and among the multiple similarities greater than the second similarity threshold, the number of similarities greater than a third similarity threshold is one, use the detection frame corresponding to the similarity greater than the third similarity threshold as a target detection frame including the target object, wherein the third similarity threshold is greater than the second similarity threshold.
[0177] In some embodiments, the determination module 5553 is further used to determine a target detection frame including the target object from at least one detection frame based on pre-stored appearance features of the target object when the detection status indicates that the target object is lost in the current frame image.
[0178] In some embodiments, the determination module 5553 is further used to extract features of objects respectively included in at least one detection frame to obtain appearance features corresponding to at least one object; respectively determine the similarity between the appearance features of each object and the appearance features of a pre-stored target object; when the number of similarities greater than a fourth similarity threshold is one, use the detection frame corresponding to the similarity greater than the fourth similarity threshold as a target detection frame including the target object; when the number of similarities greater than the fourth similarity threshold is multiple, and among the multiple similarities greater than the fourth similarity threshold, the number of similarities greater than the fifth similarity threshold is one, use the detection frame corresponding to the similarity greater than the fifth similarity threshold as a target detection frame including the target object, wherein the fifth similarity threshold is greater than the fourth similarity threshold.
[0179] In some embodiments, the appearance features of the target object are stored in a pre-created feature queue, and the target detection device also includes a storage module 5554, which is used to extract features of the target object included in the target detection box to obtain the latest appearance features of the target object; when the number of appearance features of the target object stored in the feature queue is less than the quantity threshold, the latest appearance feature is inserted to the end of the feature queue; when the number of appearance features of the target object stored in the feature queue is equal to the quantity threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted to the end of the feature queue.
[0180] In some embodiments, the determination module 5553 is also used to obtain the association status between the tracker and the detection frame associated with the target object; when the association status represents that the tracker has an associated detection frame, or when the duration of the association status representation that the tracker does not have an associated detection frame is less than a duration threshold, it is determined that the target object is not lost in the current frame image; when the association status represents that the tracker does not have an associated detection frame for a duration greater than a duration threshold, it is determined that the target object is lost in the current frame image.
[0181] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 , Figure 4 , Figure 5 , Figure 6 ,or Figure 7 The present invention should be understood by referring to the description of any one of the accompanying drawings.
[0182] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the computer device performs the target detection method described in the embodiment of the present application.
[0183] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the target detection method provided by the present application embodiment, for example, Figure 3 , Figure 4 , Figure 5 , Figure 6 ,or Figure 7 The target detection method is shown.
[0184] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM) and other memories; it may also be various devices including one or any combination of the above memories.
[0185] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0186] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
[0187] In summary, the embodiments of the present application have the following beneficial effects:
[0188] (1) The problem of poor tracking effect when the target objects intersect is improved in the scheme of using only motion information to implement the single target tracking algorithm for multi-target tracking;
[0189] (2) The problem that the target object cannot be found when it disappears from the screen is solved in the scheme of implementing a single target tracking algorithm using only motion information for multi-target tracking;
[0190] (3) The problem of the algorithm being slowed down due to the need to extract features from all objects in the picture in the solution of using motion information and appearance information to simultaneously track multiple targets to implement a single target tracking algorithm is solved.
[0191] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A target detection method, characterized in that: The method comprises: Acquire a current frame image, wherein the current frame image is a video frame image acquired at the current moment; Identify at least one detection frame from the current frame image, wherein each of the detection frames includes an object; Determining a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected among the at least one object; According to the detection state, based on the motion information of the target object and the pre-stored appearance features of the target object, or based on the pre-stored appearance features of the target object, a target detection frame including the target object is determined from the at least one detection frame.
2. The method according to claim 1, characterized in that The determining, according to the detection state, a target detection frame including the target object from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object, or based on pre-stored appearance features of the target object, comprises: When the detection state represents that the target object is not lost in the current frame image, a target detection frame including the target object is determined from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object.
3. The method according to claim 2, characterized in that The determining, based on the motion information of the target object and the pre-stored appearance features of the target object, a target detection frame including the target object from the at least one detection frame comprises: Determine a detection frame that matches the motion information of the target object from the at least one detection frame; When the number of the matched detection frames is one, performing feature extraction on the object included in the matched detection frame to obtain the appearance features of the object; Determining the similarity between the appearance features of the object and the appearance features of the target object pre-stored in the feature queue; When the similarity is greater than a first similarity threshold, the matched detection frame is determined as a target detection frame including the target object.
4. The method according to claim 3, characterized in that The feature queue stores N appearance features corresponding to the target object in the previous N frames of images, where N is a positive integer greater than 1; The determining the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue includes: Transposing a feature vector corresponding to the appearance feature of the object to obtain a transposed feature vector; Multiplying the transposed feature vector with the feature vectors corresponding to the N appearance features respectively, to obtain N first multiplication results; Subtracting the preset value from the N first multiplication results respectively, and obtaining N first subtraction results accordingly; The minimum value among the N first subtraction results is used as the similarity between the appearance feature of the object and the appearance feature of the target object pre-stored in the feature queue.
5. The method according to claim 3, characterized in that: The determining, from the at least one detection frame, a detection frame that matches the motion information of the target object comprises: Predicting the position of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image; respectively determining a distance between the at least one detection box and the predicted position; The detection frame whose distance is less than the distance threshold is used as the detection frame matched with the motion information of the target object.
6. The method according to claim 5, characterized in that The step of predicting the position of the target object based on the motion information of the target object to obtain the predicted position of the target object in the current frame image includes: Acquire motion information of the target object, wherein the motion information includes at least one of the following: a horizontal pixel position and a horizontal speed of a center of a tracking frame associated with the target object, a vertical pixel position and a vertical speed of a center of a tracking frame associated with the target object, an area and an area change speed of the tracking frame associated with the target object, and an aspect ratio of the tracking frame associated with the target object; constructing a state model based on the motion information; The state model is called to predict the position of the target object to obtain the predicted position of the target object in the current frame image.
7. The method according to claim 5, characterized in that The respectively determining the distance between the at least one detection box and the predicted position comprises: The following processing is performed for each detection frame: Subtracting the position of the detection frame from the predicted position to obtain a second subtraction result; Transposing the second subtraction result to obtain a transposed second subtraction result; Obtaining a covariance matrix between the position of the detection frame and the predicted position; Multiplying the transposed second subtraction result by the inverse matrix of the covariance matrix to obtain a second multiplication result; The second multiplication result and the second subtraction result are multiplied, and the obtained multiplication result is used as the distance between the detection box and the predicted position.
8. The method according to claim 2, characterized in that: The determining, based on the motion information of the target object and the pre-stored appearance features of the target object, a target detection frame including the target object from the at least one detection frame comprises: Determine a detection frame that matches the motion information of the target object from the at least one detection frame; When there are multiple matching detection frames, feature extraction is performed on objects respectively included in the multiple matching detection frames to obtain appearance features respectively corresponding to the multiple objects; respectively determining the similarity between the appearance feature of each of the objects and the appearance features of the target object pre-stored in the feature queue; When the number of the similarities greater than the second similarity threshold is one, the detection frame corresponding to the similarity greater than the second similarity threshold is used as a target detection frame including the target object.
9. The method according to claim 8, characterized in that The method further comprises: When there is a plurality of similarities greater than the second similarity threshold, and among the plurality of similarities greater than the second similarity threshold, the number of similarities greater than a third similarity threshold is one, a detection frame corresponding to the similarities greater than the third similarity threshold is used as a target detection frame including the target object, wherein the third similarity threshold is greater than the second similarity threshold.
10. The method according to any one of claims 1 to 9, characterized in that: The determining, according to the detection state, a target detection frame including the target object from the at least one detection frame based on motion information of the target object and pre-stored appearance features of the target object, or based on pre-stored appearance features of the target object, comprises: When the detection state indicates that the target object is lost in the current frame image, a target detection frame including the target object is determined from the at least one detection frame based on pre-stored appearance features of the target object.
11. The method according to claim 10, characterized in that The step of determining a target detection frame including the target object from the at least one detection frame based on the pre-stored appearance features of the target object comprises: Performing feature extraction on the objects respectively included in the at least one detection frame to obtain appearance features respectively corresponding to at least one of the objects; Respectively determining the similarity between the appearance features of each of the objects and the pre-stored appearance features of the target object; When the number of the similarities greater than a fourth similarity threshold is one, taking the detection frame corresponding to the similarity greater than the fourth similarity threshold as a target detection frame including the target object; When there is a plurality of similarities greater than a fourth similarity threshold, and among the plurality of similarities greater than the fourth similarity threshold, the number of similarities greater than a fifth similarity threshold is one, a detection frame corresponding to the similarities greater than the fifth similarity threshold is used as a target detection frame including the target object, wherein the fifth similarity threshold is greater than the fourth similarity threshold.
12. The method according to any one of claims 1 to 9, characterized in that: The appearance features of the target object are stored in a pre-created feature queue; After determining a target detection frame including the target object from the at least one detection frame, the method further includes: Extracting features of the target object included in the target detection frame to obtain the latest appearance features of the target object; When the number of appearance features of the target object stored in the feature queue is less than a number threshold, inserting the latest appearance feature into the tail of the feature queue; When the number of appearance features of the target object stored in the feature queue is equal to a quantity threshold, the appearance feature at the head of the feature queue is deleted, and the latest appearance feature is inserted into the tail of the feature queue.
13. The method according to any one of claims 1 to 9, characterized in that: The determining a detection state of the target object in the current frame image includes: Obtaining the association status between the tracker and the detection frame associated with the target object; When the associated state indicates that the tracker has an associated detection frame, or when the duration for which the associated state indicates that the tracker does not have an associated detection frame is less than a duration threshold, determining that the target object is not lost in the current frame image; When the association state indicates that the duration of the detection frame associated with the tracker is greater than the duration threshold, it is determined that the target object is lost in the current frame image.
14. A target detection device, characterized in that: The device comprises: An acquisition module is used to acquire a current frame image, wherein the current frame image is a video frame image acquired at a current moment; A recognition module, configured to recognize at least one detection frame from the current frame image, wherein each of the detection frames includes an object; a determination module, configured to determine a detection state of a target object in the current frame image, wherein the detection state indicates whether the target object is lost in the current frame image, and the target object is an object to be detected in the at least one object; The determination module is also used to determine a target detection frame including the target object from the at least one detection frame according to the detection state, based on motion information of the target object and pre-stored appearance features of the target object, or based on pre-stored appearance features of the target object.
15. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the target detection method according to any one of claims 1 to 13 when executing the executable instructions stored in the memory.
16. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the target detection method according to any one of claims 1 to 13 is implemented.
Citation Information
Patent Citations
Target tracking method and device, electronic equipment and computer readable storage medium
CN111640140A
Multi-target tracking method and device, electronic equipment and readable storage medium
CN112907636A
Target tracking method, device, computer equipment and storage medium
CN113034541A
Pedestrian tracking method and device based on pedestrian re-identification, storage medium and equipment
CN114332169A
Multi-target tracking method, device, equipment and medium
CN114820699A
Cited By
Target detection method and apparatus, and electronic device, computer-readable storage medium and computer program product
WO2026020739A1
