Target object moving trajectory tracking method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a moving trajectory tracking method and device of a target object, electronic equipment and a medium, first coordinate information of the target object in a target image is acquired, the target image is an image of a target space acquired by a target camera, and the moving trajectory of the target object is tracked based on the first coordinate information and a perspective transformation matrix. Acquiring second coordinate information of the target object in the target space, and determining a moving track of the target object based on the second coordinate information, that is, converting the first coordinate information of the target object in the target image into standardized second coordinate information in combination with a perspective transformation matrix, and then the moving track of the target object is tracked according to the second coordinate information of the target object, so that moving track tracking of the target object under the standard coordinate is realized, and meanwhile, the accuracy of the determined moving track is improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent recognition technology, and specifically relates to a method, device, electronic device and medium for tracking the movement trajectory of a target object. Background Art

[0002] In the field of intelligent recognition, by tracking the target object's activity trajectory, recording and analyzing the target object's location information during the recognition process, the target object's behavior can be optimized and guided. For example, in a teaching scenario, by tracking the teacher's mobile position, the teacher's teaching behavior can be optimized.

[0003] In the prior art, tracking and recording the activity trajectory of a target object mainly relies on a video capture device, which captures an image of a scene, then identifies the target object in the image and records its coordinates within the scene.

[0004] However, using the existing technology, the accuracy of determining the activity trajectory of the target object is not high. Summary of the invention

[0005] Embodiments of the present application provide a method, device, electronic device, and medium for tracking a movement trajectory of a target object to solve the problem of low accuracy of the determined movement trajectory in the prior art.

[0006] A first aspect of an embodiment of the present application provides a method for tracking a moving trajectory of a target object, comprising: Acquire first coordinate information of a target object in a target image, where the target image is an image of a target space acquired by a target camera; Based on the first coordinate information and the perspective transformation matrix, obtaining second coordinate information of the target object in the target space; A moving trajectory of the target object is determined based on the second coordinate information.

[0007] Optionally, before acquiring second coordinate information of the target object in the target space based on the first coordinate information and the perspective transformation matrix, the method further includes: Acquire four source points of the first image of the target space, where the four source points are four vertices of the first image; The perspective transformation matrix is ​​obtained based on four source points of the first image and four target points of the target space.

[0008] Optionally, the acquiring four source points of the first image of the target space includes: Obtain the original image of the target space captured by the target camera; Performing mask processing on the desk and chair area in the original image to obtain a mask image; Binarizing the mask image to obtain a binary image; Perform edge detection on the binary image to obtain a polygonal image; If the polygonal image is a quadrilateral, determining the polygonal image as the first image; If the polygonal image is a hexagon, the hexagon is preprocessed to obtain a quadrilateral corresponding to the hexagon, and the quadrilateral is determined as the first image.

[0009] Optionally, preprocessing the hexagon to obtain a quadrilateral corresponding to the hexagon includes: Extending the first hypotenuse, the second hypotenuse, and the base of the hexagon to obtain a first intersection point of an extension line of the first hypotenuse and an extension line of the base, and a second intersection point of an extension line of the second hypotenuse and an extension line of the base; Determine that a quadrilateral formed by two endpoints of the top side of the hexagon, the first intersection point, and the second intersection point is the quadrilateral corresponding to the hexagon.

[0010] Optionally, performing mask processing on the desk area in the original image to obtain a mask image includes: The original image is input into a first preset model, the desk and chair area in the original image is identified, and the desk and chair area is masked, wherein the first preset model is trained based on sample images and sample annotations, and the sample images include annotated desks and chairs.

[0011] Optionally, before obtaining the first coordinate information of the target object in the target image, the method further includes: The target object is identified based on the body features of the target object.

[0012] Optionally, identifying the target object based on the human features of the target object includes: Inputting the human body picture of the target object into a second preset model to obtain a first human body feature vector, wherein the second preset model is obtained based on cross-domain data pre-training; Adding the first human body feature vector to a human body feature list; Performing human body detection on each video frame, extracting features of the human body in the first posture, and obtaining a second human body feature vector; The target object is identified based on the similarity value between the second human feature vector and each first human feature vector in the human feature list.

[0013] Optionally, identifying the target object based on a similarity value between the second human feature vector and each first human feature vector in the human feature list includes: A similarity value between the second human feature vector and each first human feature vector is obtained, and a maximum value among the similarity values ​​is determined to be greater than a first preset threshold value, and then the human body corresponding to the second human feature vector is determined to be the target object.

[0014] Optionally, also include: If the maximum value of the similarity values ​​is greater than a second preset threshold, the second human feature vector is added to the human feature list, and the second preset threshold is greater than the first preset threshold.

[0015] Optionally, also include: If the number of first human feature vectors in the human feature list is greater than a preset threshold, one or more of the first human feature vectors are deleted.

[0016] Optionally, before identifying the target object based on the human features of the target object, the method further includes: Based on face detection, identify the face of the target object in the video frame and obtain a human body image of the target object; or, It is determined that among multiple human bodies in a first posture in the video, a human body in the first posture for the longest time is a target object, and a human body image of the target object is obtained.

[0017] A second aspect of an embodiment of the present application provides a moving trajectory tracking device for a target object, comprising: An acquisition module, used to acquire first coordinate information of a target object in a target image, wherein the target image is an image of a target space acquired by a target camera; A processing module, configured to obtain second coordinate information of the target object in the target space based on the first coordinate information and a perspective transformation matrix; A tracking module is used to determine the moving trajectory of the target object based on the second coordinate information.

[0018] Optionally, the processing module is specifically used to obtain four source points of the first image of the target space, where the four source points are four vertices of the first image; and obtain the perspective transformation matrix based on the four source points of the first image and the four target points of the target space.

[0019] Optionally, the processing module is specifically used to obtain an original image of the target space captured by a target camera; perform mask processing on the desk and chair area in the original image to obtain a mask image; perform binarization processing on the mask image to obtain a binarized image; perform edge detection on the binarized image to obtain a polygonal image; if the polygonal image is a quadrilateral, determine the polygonal image as the first image; if the polygonal image is a hexagon, pre-process the hexagon to obtain a quadrilateral corresponding to the hexagon, and determine the quadrilateral as the first image.

[0020] Optionally, the processing module is specifically used to extend the first hypotenuse, the second hypotenuse and the base of the hexagon to obtain a first intersection point of the extension line of the first hypotenuse and the extension line of the base, and a second intersection point of the extension line of the second hypotenuse and the extension line of the base; and determine that the quadrilateral formed by the two endpoints of the top side of the hexagon, the first intersection point and the second intersection point is the quadrilateral corresponding to the hexagon.

[0021] Optionally, the processing module specifically inputs the original image into a first preset model, identifies the desk and chair area in the original image, and performs mask processing on the desk and chair area, wherein the first preset model is trained based on sample images and sample annotations, and the sample image includes annotated desks and chairs.

[0022] Optionally, the acquisition module is specifically used to identify the target object based on the human body features of the target object.

[0023] Optionally, the acquisition module is specifically used to input a human body picture of the target object into a second preset model to obtain a first human body feature vector, wherein the second preset model is obtained based on cross-domain data pre-training; add the first human body feature vector to a human body feature list; perform human body detection on each video frame, perform feature extraction on a human body in a first posture, and obtain a second human body feature vector; and identify the target object based on similarity values ​​between the second human body feature vector and each first human body feature vector in the human body feature list.

[0024] Optionally, the acquisition module is specifically used to obtain a similarity value between the second human feature vector and each first human feature vector, and determine that if a maximum value among the similarity values ​​is greater than a first preset threshold, then determine that the human body corresponding to the second human feature vector is the target object.

[0025] Optionally, the acquisition module is specifically used to add the second human feature vector to the human feature list if the maximum value of the similarity values ​​is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold.

[0026] Optionally, the acquisition module is specifically configured to delete one or more first human feature vectors if the number of first human feature vectors in the human feature list is greater than a preset threshold.

[0027] Optionally, the acquisition module is specifically used to identify the face of the target object in the video frame based on face detection, and acquire a human body picture of the target object; or, It is determined that among multiple human bodies in a first posture in the video, a human body in the first posture for the longest time is a target object, and a human body image of the target object is obtained.

[0028] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for tracking the movement trajectory of the target object described in the first aspect are implemented.

[0029] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for tracking the movement trajectory of the target object described in the first aspect are implemented.

[0030] A fifth aspect of an embodiment of the present application provides a computer program product, which, when executed by a processor of a vehicle or a cloud server, implements the steps of the method for tracking the movement trajectory of a target object described in the first aspect above.

[0031] The embodiments of the present application provide a method, device, electronic device and medium for tracking the movement trajectory of a target object. The method obtains first coordinate information of the target object in a target image, where the target image is an image of a target space acquired by a target camera. Based on the first coordinate information and a perspective transformation matrix, second coordinate information of the target object in the target space is obtained. Based on the second coordinate information, the movement trajectory of the target object is determined. That is, the first coordinate information of the target object in the target image is combined with the perspective transformation matrix to be converted into standardized second coordinate information. Thus, the position deviation problem caused by image distortion is overcome and more accurate coordinate information can be obtained. Then, the movement trajectory of the target object is tracked according to the second coordinate information of the target object. Thus, the movement trajectory tracking of the target object under standard coordinates is realized, and the accuracy of the determined movement trajectory is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a flow chart of a method for tracking a moving trajectory of a target object provided in an embodiment of the present application; Figure 2AA schematic diagram of a scene in which a target object is located in a target space with a first coordinate provided in an embodiment of the present application; Figure 2B A schematic diagram of a scene in which a target object is located in a standard target space with a second coordinate provided in an embodiment of the present application; Figure 3 A schematic diagram of a flow chart of another method for tracking the movement trajectory of a target object provided in an embodiment of the present application; Figure 4 A schematic diagram of a flow chart of another method for tracking the movement trajectory of a target object provided in an embodiment of the present application; Figure 5A A schematic diagram of a scene of a binary mask image provided in an embodiment of the present application; Figure 5B A schematic diagram of a scene of a binary mask image with filled edges provided in an embodiment of the present application; Fig. 6A A schematic diagram of a scene with a polygonal outline provided in an embodiment of the present application; Figure 6B A schematic diagram of a polygonal image scene provided in an embodiment of the present application; Fig. 7A A schematic diagram of a scene of a quadrilateral polygonal image provided in an embodiment of the present application; Figure 7B A schematic diagram of a scene of a hexagonal polygonal image provided in an embodiment of the present application; Figure 8 A schematic diagram of a flow chart of another method for tracking a moving trajectory of a target object provided in an embodiment of the present application; Fig. 9 A schematic diagram of a flow chart of another method for tracking a moving trajectory of a target object provided in an embodiment of the present application; Fig.10 A schematic diagram of a flow chart of another method for tracking a moving trajectory of a target object provided in an embodiment of the present application; Fig.11 A schematic diagram of the structure of a moving trajectory tracking device for a target object provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results, etc. in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result, etc. according to the judgment result.

[0036] In view of the problem that the existing technology is affected by factors such as the camera deployment position, the captured image is distorted compared with the actual scene, resulting in the problem that the accuracy of the determined movement trajectory is not high, the present application provides a method for tracking the movement trajectory of a target object, by combining the first coordinate information of the target object in the target image with a perspective transformation matrix, converting the second coordinate information, and then tracking the movement trajectory of the target object according to the second coordinate information of the target object, thereby realizing the movement trajectory tracking of the target object under standard coordinates, and at the same time improving the accuracy of the determined movement trajectory.

[0037] In the embodiment of the present application, the coordinate information of the target object in the target image is first obtained, and then the coordinate information of the target object is transformed by combining the perspective transformation matrix. For the convenience of description, in the following embodiments of the present application, the coordinate information of the target object in the target image is described as the first coordinate information, and the coordinate information obtained by the first coordinate information and the perspective transformation matrix is ​​described as the second coordinate information. The quadrilateral corresponding to the acquired desk and chair area is described as the first image, the upper left hypotenuse of the hexagon corresponding to the desk and chair area is described as the first hypotenuse, the upper right hypotenuse of the hexagon corresponding to the desk and chair area is described as the second hypotenuse, the intersection of the extension line of the first hypotenuse and the extension line of the bottom of the hexagon is described as the first intersection, and the intersection of the extension line of the second hypotenuse and the extension line of the bottom of the hexagon is described as the second intersection. The segmentation model for identifying and masking the desk and chair area is described as the first preset model, and the human body recognition model for extracting human body features from the human body is described as the second preset model. The human body feature vector of the extracted target object is described as the first human body feature vector, the standing posture is described as the first posture, and the human body feature vector extracted from the human body in the first posture of each video frame is described as the second human body feature vector. The maximum similarity value between the second human feature vector and the first human feature vector capable of determining that the human body in the first posture is the target object is described as a first preset threshold, and the similarity value capable of adding the second human feature to the human feature list is described as a second preset threshold.

[0038] The identification and tracking of the target object in the technical solution of the present application is carried out on the premise of authorization by the target object.

[0039] The following describes the technical solution of the target object movement trajectory tracking method of the present application by taking several embodiments as examples: Figure 1 A flow chart of a method for tracking the movement trajectory of a target object provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method of this embodiment is as follows: S11: Acquire first coordinate information of a target object in a target image, where the target image is an image of a target space acquired by a target camera.

[0040] The target image is an image of the target space acquired by the target camera, the target object is a human body whose movement trajectory is to be tracked in the target image, and the first coordinate information is the coordinate information of the human body whose movement trajectory is to be tracked in the target image. For example, when the target space is a classroom and the target object is a teacher, to track the movement trajectory of the teacher, the image of the entire classroom space must first be acquired through the camera in the classroom, and then the coordinate information of the teacher in the image must be determined.

[0041] S12: Based on the first coordinate information and the perspective transformation matrix, obtain second coordinate information of the target object in the target space.

[0042] Among them, due to the perspective problem of the target camera, the image of the target space obtained by the target camera may be distorted, especially in the case of a wide-angle lens or a high camera installation position. Perspective distortion will cause the human body far away from the target camera to appear smaller in the image, and the human body at a close distance to appear larger, thereby affecting the accurate coordinate calculation. Therefore, it is necessary to obtain the perspective transformation matrix of the target image, convert the coordinate system of the target image into the standard coordinate system, and obtain the coordinate information of the target object in the standard coordinate system, that is, the second coordinate information, according to the product of the first coordinate information of the target object in the target image and the perspective transformation matrix.

[0043] For example, when the target space is a classroom, the target object is a teacher. By calculating the product of the teacher's coordinate information in the image and the perspective transformation matrix of the standardized classroom space image, the actual standard coordinate information of the teacher in the standardized classroom space is obtained, where: Figure 2A A schematic diagram of a scene in which a target object is located in a target space with a first coordinate provided in an embodiment of the present application, Figure 2B A schematic diagram of a scene in which a target object is located in a standard target space with a second coordinate provided in an embodiment of the present application.

[0044] S13: Determine a moving trajectory of the target object based on the second coordinate information.

[0045] Optionally, after obtaining the second coordinate information of the target object, the moving trajectory of the target object in the standardized target space can be obtained. For example, when the target space is a classroom, the target object is a teacher, and the standard coordinates of the teacher in the standardized classroom space are obtained. Then, the moving trajectory of the teacher in the standardized classroom space can be tracked by the target camera.

[0046] In an embodiment of the present application, first coordinate information of a target object in a target image is obtained, where the target image is an image of a target space acquired by a target camera, and second coordinate information of the target object in the target space is obtained based on the first coordinate information and a perspective transformation matrix, and a moving trajectory of the target object is determined based on the second coordinate information. That is, the first coordinate information of the target object in the target image is converted into second coordinate information in combination with the perspective transformation matrix, and then the moving trajectory of the target object is tracked according to the second coordinate information of the target object, thereby achieving tracking of the moving trajectory of the target object in standard coordinates and improving the accuracy of the determined moving trajectory.

[0047] Figure 3A flow chart of another method for tracking the movement trajectory of a target object provided in an embodiment of the present application, Figure 3 is Figure 1 Based on the embodiment shown, a possible implementation method of obtaining the perspective transformation matrix is ​​further described, as follows: Figure 3 As shown, the method of this embodiment is as follows: S31: Acquire four source points of the first image of the target space, where the four source points are four vertices of the first image.

[0048] Among them, the first image of the target space is a quadrilateral corresponding to the desk and chair area obtained by the target camera when the target space is a classroom. The four source points of the first image are the four vertices of the quadrilateral, and then the coordinates of the four source points are obtained.

[0049] S32: Acquire the perspective transformation matrix based on the four source points of the first image and the four target points of the target space.

[0050] Among them, due to the viewing angle problem of the target camera, the obtained quadrilateral is not a regular rectangle, and the four target points of the target space are the four vertices of the standard rectangle. Then the coordinates of the four target points are obtained, and then the perspective transformation matrix is ​​calculated according to the coordinate matrix of the four source points of the first image and the coordinate matrix of the four target points of the standard rectangle, and then the coordinate system of the target space is converted to the standard coordinate system. The perspective transformation matrix calculation is expressed as M=A −1 B, where A is the source matrix, A -1 is the inverse matrix of the source point matrix, and B is the target point matrix.

[0051] In this embodiment, by acquiring four source points of the first image of the target space, the four source points are four vertices of the first image, and based on the four source points of the first image and the four target points of the target space, the perspective transformation matrix is ​​acquired. Thus, the perspective transformation matrix of the first image is obtained, and the accuracy of the coordinates of the target object in the target space is improved.

[0052] Figure 4 A flowchart of another method for tracking the movement trajectory of a target object provided in an embodiment of the present application is shown in FIG. Figure 4 is Figure 3 Based on the embodiment shown, a possible implementation of S31 is further described as follows: Figure 4 As shown, the method of this embodiment is as follows: S311: Acquire an original image of the target space captured by the target camera.

[0053] Wherein, when the target space is a classroom, the original image of the target space captured by the target camera includes the desk and chair area of ​​the classroom.

[0054] S312: Masking the desk and chair area in the original image to obtain a masked image.

[0055] Optionally, the original image is input into a first preset model, the desk and chair area in the original image is identified, and the desk and chair area is masked, wherein the first preset model is trained based on sample images and sample annotations, and the sample images include annotated desks and chairs.

[0056] Among them, the first preset model is the eighth generation YOLO instance segmentation model (YOLOv8 SegmentationModel, YOLOv8-Seg), the sample image is an image containing the desk and chair area when the target space is a classroom, the sample annotation is the annotation of the desk and chair area in the sample image, and the specific operation is to mark the polygon annotation corner points of the desk and chair area, and then use the sample image and sample annotation to further train the YOLOv8-seg model, and then the original image is input into the YOLOv8-seg model, the YOLOv8-seg model recognizes the desk and chair area of ​​the original image, and performs mask processing on the desk and chair area to obtain a mask image of the desk and chair area.

[0057] S313: Binarize the mask image to obtain a binary image.

[0058] Among them, Figure 5A and Figure 5B As shown in the figure, it is a schematic diagram of the scene of the binary mask image provided by the embodiment of the present application. First, the mask image of the desk and chair area is binarized to obtain the binary mask image of the desk and chair area. In addition, in order to facilitate the subsequent detection of the complete outline, the binary mask image is filled with the background color at the edge, 50 pixels at the top and bottom, and 2 pixels at the left and right. Among them, Figure 5A A schematic diagram of a scene of a binary mask image provided in an embodiment of the present application, Figure 5B A schematic diagram of a scene of a binary mask image with filled edges provided in an embodiment of the present application.

[0059] S314: Perform edge detection on the binary image to obtain a polygonal image.

[0060] The edge detection method is the canny edge detection method, and the obtained binary mask image of the filled edge is subjected to canny edge detection to obtain the polygonal contour of the desk and chair area, and all contours are detected, the contour with the largest area is taken, and the polygonal image approximating the desk and chair area is obtained using the polygonal approximation method. Fig. 6A A schematic diagram of a scene with a polygonal outline provided in an embodiment of the present application, Figure 6B A schematic diagram of a polygonal image scene provided in an embodiment of the present application.

[0061] S315: Determine whether the polygonal image is a quadrilateral. If it is a quadrilateral, execute S316; if it is not a quadrilateral, execute S317.

[0062] S316: The quadrilateral image is the first image.

[0063] If the polygonal image approximating the desk and chair area is determined to be a quadrilateral, the quadrilateral image is determined to be the first image, and then the four vertices of the quadrilateral are determined as the four source points of the first image, and finally the coordinates of the four source points of the first image are determined. Fig. 7A A schematic diagram of a scene of a quadrilateral polygonal image provided in this application, such as Fig. 7A As shown, A1, A2, B1, and B2 are four vertices of the quadrilateral.

[0064] S317: If the polygonal image is a hexagon, pre-process the hexagon to obtain a quadrilateral corresponding to the hexagon, and determine the quadrilateral as the first image.

[0065] If the polygonal image approximating the desk and chair area is determined to be a hexagon, the hexagon is preprocessed, and the hexagon is completed into a quadrilateral by calculating the intersection of straight lines, and the completed quadrilateral is determined to be the first image, and then the four vertices of the completed quadrilateral are determined as the four source points of the first image, and finally the coordinates of the four source points of the first image are determined. Figure 7B A schematic diagram of a scene of a hexagonal polygonal image provided in this application, such as Figure 7B As shown, A1, A2, B1, B2, C1, and C2 are six vertices of the hexagon, and the four vertices of the quadrilateral are A2, B1, D1, and D2 which are completed by calculating the intersection points of the straight lines.

[0066] In this embodiment, an original image of a target space captured by a target camera is obtained, a mask processing is performed on the desk and chair area in the original image to obtain a mask image, a binarization processing is performed on the mask image to obtain a binarized image, and edge detection is performed on the binarized image to obtain a polygonal image. If the polygonal image is a quadrilateral, the polygonal image is determined as the first image. If the polygonal image is a hexagon, the hexagon is pre-processed to obtain a quadrilateral corresponding to the hexagon, and the quadrilateral is determined as the first image. Thus, the coordinates of the four source points of the first image are determined, thereby facilitating more accurate acquisition of the perspective transformation matrix of the first image.

[0067] Figure 8 A flow chart of another method for tracking the moving trajectory of a target object provided in an embodiment of the present application is provided. Figure 8 is Figure 4 Based on the embodiment shown, a possible implementation of S317 is further described as follows: Figure 8 As shown, the method of this embodiment is as follows: S3171: Extend the first hypotenuse, the second hypotenuse and the base of the hexagon to obtain a first intersection point between the extension line of the first hypotenuse and the extension line of the base, and a second intersection point between the extension line of the second hypotenuse and the extension line of the base.

[0068] Among them, Figure 7B As shown, the first hypotenuse of the hexagon is the line connecting points A1 and A2, the second hypotenuse of the hexagon is the line connecting points B1 and B2, and the base of the hexagon is the line connecting points C1 and C2. The line connecting points A1 and A2, the line connecting points B1 and B2, and the line connecting points C1 and C2 are extended and the equations of the straight lines are calculated. The first intersection point is the intersection point D1 of the straight line where points A1 and A2 are located and the straight line where points C1 and C2 are located. The second intersection point is the intersection point D2 of the straight line where points B1 and B2 are located and the straight line where points C1 and C2 are located.

[0069] S3172: Determine that a quadrilateral formed by two endpoints of the top side of the hexagon, the first intersection point, and the second intersection point is the quadrilateral corresponding to the hexagon.

[0070] Among them, Figure 7B As shown, the two endpoints of the top side of the hexagon are A2 and B1, the first intersection point is D1, and the second intersection point is D2, so the quadrilateral formed by the four points A2, B1, D1, and D2 is the quadrilateral corresponding to the hexagon.

[0071] In this embodiment, the first hypotenuse, the second hypotenuse and the base of the hexagon are extended to obtain a first intersection point of the extension line of the first hypotenuse and the extension line of the base, and a second intersection point of the extension line of the second hypotenuse and the extension line of the base; the quadrilateral formed by the two endpoints of the top side of the hexagon, the first intersection point and the second intersection point is determined to be the quadrilateral corresponding to the hexagon, thereby achieving approximation of the desk and chair area to a quadrilateral, which facilitates the perspective transformation of the desk and chair area to a rectangle.

[0072] Fig. 9 A flow chart of another method for tracking the moving trajectory of a target object provided in an embodiment of the present application is provided. Fig. 9 A description of a possible implementation method of identifying the target object based on the human features of the target object before obtaining the first coordinate information of the target object in the target image as described in the present application, such as Fig. 9 As shown, the method of this embodiment is as follows: S91: Inputting the human body picture of the target object into a second preset model to obtain a first human body feature vector, wherein the second preset model is obtained based on cross-domain data pre-training.

[0073] The second preset model is an omni-scale network (OSNet) pre-trained based on cross-domain data, which is used to capture human features of different scales. Pre-training based on cross-domain data can ensure the generalization ability of the model and ensure the application of the model on multiple target cameras. The human body image of the target object is obtained from the video frame taken by the target camera, and the human body image of the target object is input into the OSNet pre-trained based on cross-domain data, and the human body feature vector of the target object extracted is the first human body feature vector.

[0074] S92: Add the first human body feature vector to a human body feature list.

[0075] Among them, first, a human feature list of a target object is created, and then the first human feature vector of the target object extracted from the human picture of the target object is added to the human feature list. Each element in the list is a feature vector. At this time, the length of the list is 1, indicating that the number of human features of the target object is 1.

[0076] S93: Perform human body detection on each video frame, extract features of the human body in the first posture, and obtain a second human body feature vector.

[0077] Among them, the first posture is a standing posture. Since the subsequent video frames of the video acquired by the target camera may contain target objects and other human bodies, when the target space is a classroom, the target object is a teacher, the teacher is in a standing posture, and the student is in a sitting posture. Based on this, the human bodies of teachers and students can be distinguished. That is, in order to determine the identity of the teacher, it is necessary to detect the human body in a standing posture. The model for human body detection is a YOLOv8 model. Each video frame is input into the YOLOv8 model, and human body detection is performed on each standing human body in the video frame. All human bodies in a standing posture are detected and marked with human body frames. Then, each video frame is input into OSNet based on cross-domain data pre-training, and the human body feature vector extracted from the human body in a standing posture in each video frame is the second human body feature vector.

[0078] S94: Identify the target object based on the similarity value between the second human feature vector and each first human feature vector in the human feature list.

[0079] Optionally, a similarity value between the second human feature vector and each first human feature vector is obtained, and a maximum value among the similarity values ​​is determined to be greater than a first preset threshold, and then the human body corresponding to the second human feature vector is determined to be the target object.

[0080] Optionally, if the maximum value of the similarity values ​​is greater than a second preset threshold, the second human feature vector is added to the human feature list, and the second preset threshold is greater than the first preset threshold.

[0081] Optionally, if the number of first human feature vectors in the human feature list is greater than a preset threshold, one or more of the first human feature vectors are deleted.

[0082] Among them, the initial length of the human feature list is 1, that is, the number of human features of the target object is 1, and the cosine similarity calculation is performed on the second human feature vectors of all human bodies in a standing posture in the video frame and the first human feature vector of the target object to obtain the similarity value between each second human feature vector and the first human feature vector. Among them, cosine similarity is a measure of the similarity between two non-zero vectors. The similarity between the two vectors is determined by measuring the cosine value of the angle between them. Usually, the cosine similarity of human features ranges from 0 to 1, 0 means that the two vectors are orthogonal, that is, there is no similarity between them, and 1 means that the two vectors are exactly the same. Assuming that the first human feature vector is X and the second human feature vector is Y, the cosine similarity between the two is expressed as ,in, is the Euclidean norm of vectors A and B, i.e., the length. Then, for the calculated cosine similarity value, a first preset threshold and a second preset threshold are preset, wherein the second preset threshold is greater than the first preset threshold. When the similarity value of the second human feature of a standing human body and the first human feature of the target object is greater than the first preset threshold, the standing human body is judged to be the target object. When the similarity value is greater than the second preset threshold, the second human feature vector of the human body detected in this frame is added to the human feature list as the first human feature vector. At this time, a preset threshold is preset for the number of first human feature vectors in the human feature list. If the number of first human feature vectors in the human feature list is greater than the preset threshold, that is, when one or more new first human feature vectors are added, one or more old first human feature vectors are randomly deleted.

[0083] For example, when the target space is a classroom and the target object is a teacher, the cosine similarity between the human features of all standing people in the video frame and the human features of the teacher is calculated, assuming that the first preset threshold is 0.9850 and the second preset threshold is 0.9950; for the human feature vector of the teacher in the teacher human feature list, the preset threshold is 30; assuming that the human feature vector of the teacher is [2,3,4], assuming that the human feature vector of the standing person with the highest similarity to the human feature of the teacher in the video frame is [2,2,4], the cosine similarity between them is is approximately 0.9855, which is greater than 0.9850, then the standing person can be judged as the teacher; assuming that the teacher's body feature vector is [2,3,4], and assuming that the body feature vector of the standing person with the highest similarity to the teacher's body features in the video frame is [1.5,3,4], and the cosine similarity value between them is approximately 0.9960, which is greater than 0.9950, then the body feature vector of the standing person is added to the teacher's body feature list. If there are already 30 body feature vectors of the teacher in the teacher's body feature list at this time, a vector is randomly selected for replacement.

[0084] In this embodiment, a first human feature vector is obtained by inputting a human body picture of the target object into a second preset model, wherein the second preset model is obtained based on cross-domain data pre-training, the first human feature vector is added to a human body feature list, human body detection is performed on each video frame, and features are extracted from a human body in a first posture to obtain a second human feature vector, and based on similarity values ​​between the second human feature vector and each first human feature vector in the human feature list, the target object is identified, thereby realizing multi-camera tracking of the target object's moving trajectory, and facilitating the acquisition of the first coordinate information of the target object.

[0085] Fig.10A flow chart of another method for tracking the movement trajectory of a target object provided in an embodiment of the present application is a description of a possible implementation method of obtaining a human body picture of the target object before identifying the target object based on the human body features of the target object, such as Fig.10 As shown, the method of this embodiment is as follows: S101: Based on face detection, recognize the face of a target object in a video frame.

[0086] Among them, after obtaining the video containing the target object through the target camera, it is necessary to perform face detection on the face of the video frame to determine the identity of the target object in the video frame. The face detection model is YOLOv7. First, the YOLOv7 model is trained with sample images and sample annotations containing the face of the target object, and then the video frame is input into the trained YOLOv7 model to detect the face in the video frame and mark the face frame.

[0087] S102: Determine whether the face of the target object in the video frame can be recognized. If it can be recognized, execute S103; if it cannot be recognized, execute S104.

[0088] S103: Identify the face of the target object in the video frame, and obtain a human body image of the target object.

[0089] Among them, since there may be a target object and other human bodies in the video frame, when the target space is a classroom, the target object is a teacher, the teacher is in a standing posture, and the student is in a sitting posture. Based on this, the human bodies of the teacher and the student can be distinguished. That is, in order to determine the identity of the teacher, it is necessary to detect the human body in the standing posture. Therefore, it is also necessary to detect the human body in the first posture, that is, standing, in the video frame through a human body detection model. The human body detection model is YOLOv8. The video frame is input into the YOLOv8 model, and the human body standing in the video frame is identified and the human body frame is marked.

[0090] If the face of the target object in the video frame is recognized, the corresponding human frame is matched according to the intersection over union (IOU) to determine the human body of the target object, and finally obtain the human body image of the target object. The intersection over union is used to evaluate the degree of overlap between two regions. It is the ratio of the area of ​​the intersection of the two regions to the area of ​​their union. Assuming that the face frame area of ​​the target object is M and the corresponding human frame area is N, the intersection over union ratio between them is expressed as .

[0091] S104: Determine, among multiple human bodies in the first posture in the video, that the human body in the first posture for the longest time is a target object, and obtain a human body image of the target object.

[0092] Among them, if the face of the target object in the video frame cannot be recognized, the YOLOv8 model is used to detect the human body that has stood the longest among the standing human bodies in the video frame, and it is determined to be the human body of the target object, and finally the human body image of the target object is obtained.

[0093] In this embodiment, the face of the target object in the video frame is identified based on face detection. If the face can be identified, the body picture of the target object is obtained. If the face cannot be identified, the body that has been in the first posture for the longest time among multiple bodies in the first posture in the video is determined to be the target object, and the body picture of the target object is obtained, thereby achieving identity distinction between the target object and other bodies in the target space.

[0094] Fig.11 A schematic diagram of a moving trajectory tracking device for a target object provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the device of this embodiment includes: an acquisition module 1101, a processing module 1102 and a tracking module 1103, wherein the acquisition module 1101 is used to obtain the first coordinate information of the target object in the target image, and the target image is an image of the target space acquired by the target camera.

[0095] The processing module 1102 is used to obtain second coordinate information of the target object in the target space based on the first coordinate information and the perspective transformation matrix.

[0096] The tracking module 1103 is configured to determine a moving trajectory of the target object based on the second coordinate information.

[0097] Optionally, the processing module 1102 is specifically used to obtain four source points of the first image of the target space, where the four source points are four vertices of the first image; and obtain the perspective transformation matrix based on the four source points of the first image and the four target points of the target space.

[0098] Optionally, the processing module 1102 is specifically used to obtain an original image of the target space captured by a target camera; perform mask processing on the desk and chair area in the original image to obtain a mask image; perform binarization processing on the mask image to obtain a binarized image; perform edge detection on the binarized image to obtain a polygonal image; if the polygonal image is a quadrilateral, the polygonal image is determined as the first image; if the polygonal image is a hexagon, the hexagon is pre-processed to obtain a quadrilateral corresponding to the hexagon, and the quadrilateral is determined as the first image.

[0099] Optionally, the processing module 1102 is specifically used to extend the first hypotenuse, the second hypotenuse and the base of the hexagon to obtain a first intersection point between the extension line of the first hypotenuse and the extension line of the base, and a second intersection point between the extension line of the second hypotenuse and the extension line of the base; and determine that the quadrilateral formed by the two endpoints of the top side of the hexagon, the first intersection point and the second intersection point is the quadrilateral corresponding to the hexagon.

[0100] Optionally, the processing module 1102 is specifically used to input the original image into a first preset model, identify the desk and chair area in the original image, and perform mask processing on the desk and chair area, wherein the first preset model is trained based on sample images and sample annotations, and the sample image includes annotated desks and chairs.

[0101] Optionally, the acquisition module 1101 is specifically used to identify the target object based on the human body features of the target object.

[0102] Optionally, the acquisition module 1101 is specifically used to input a human body picture of the target object into a second preset model to obtain a first human body feature vector, wherein the second preset model is obtained based on cross-domain data pre-training; add the first human body feature vector to a human body feature list; perform human body detection on each video frame, perform feature extraction on a human body in a first posture, and obtain a second human body feature vector; and identify the target object based on similarity values ​​between the second human body feature vector and each first human body feature vector in the human body feature list.

[0103] Optionally, the acquisition module 1101 is specifically used to obtain similarity values ​​between the second human feature vector and each first human feature vector, and determine that if the maximum value of the similarity values ​​is greater than a first preset threshold, then determine that the human body corresponding to the second human feature vector is the target object.

[0104] Optionally, the acquisition module 1101 is specifically used to add the second human feature vector to the human feature list if the maximum value of the similarity values ​​is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold.

[0105] Optionally, the acquisition module 1101 is specifically configured to delete one or more first human feature vectors if the number of the first human feature vectors in the human feature list is greater than a preset threshold.

[0106] Optionally, the acquisition module 1101 is specifically used to identify the face of the target object in the video frame based on face detection, and obtain a human body picture of the target object; or, It is determined that among multiple human bodies in a first posture in the video, a human body in the first posture for the longest time is a target object, and a human body image of the target object is obtained.

[0107] The device of this embodiment corresponds to the technical solution that can be used to execute the above-mentioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0108] An embodiment of the present application also provides an electronic device, including: a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of any of the above-mentioned embodiments of the method for tracking the movement trajectory of the target object are implemented.

[0109] An embodiment of the present application further provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of any of the above-mentioned embodiments of the method for tracking the movement trajectory of a target object are implemented.

[0110] An embodiment of the present application also provides a computer program product, which, when executed by a processor of a vehicle or a cloud server, implements the steps of any of the above-mentioned embodiments of the method for tracking the movement trajectory of a target object.

[0111] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable the terminal or network side device to execute the methods described in each embodiment of the present application.

[0113] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A method for tracking a moving trajectory of a target object, characterized in that: include: Acquire first coordinate information of a target object in a target image, where the target image is an image of a target space acquired by a target camera; Based on the first coordinate information and the perspective transformation matrix, obtaining second coordinate information of the target object in the target space; A moving trajectory of the target object is determined based on the second coordinate information.

2. The method according to claim 1, characterized in that Before acquiring the second coordinate information of the target object in the target space based on the first coordinate information and the perspective transformation matrix, the method further includes: Acquire four source points of the first image of the target space, where the four source points are four vertices of the first image; The perspective transformation matrix is ​​obtained based on four source points of the first image and four target points of the target space.

3. The method according to claim 2, characterized in that The acquiring of four source points of the first image of the target space comprises: Obtain the original image of the target space captured by the target camera; Performing mask processing on the desk and chair area in the original image to obtain a mask image; Binarizing the mask image to obtain a binary image; Perform edge detection on the binary image to obtain a polygonal image; If the polygonal image is a quadrilateral, determining the polygonal image as the first image; If the polygonal image is a hexagon, the hexagon is preprocessed to obtain a quadrilateral corresponding to the hexagon, and the quadrilateral is determined as the first image.

4. The method according to claim 3, characterized in that The preprocessing of the hexagon to obtain a quadrilateral corresponding to the hexagon includes: Extending the first hypotenuse, the second hypotenuse, and the base of the hexagon to obtain a first intersection point of an extension line of the first hypotenuse and an extension line of the base, and a second intersection point of an extension line of the second hypotenuse and an extension line of the base; Determine that a quadrilateral formed by two endpoints of the top side of the hexagon, the first intersection point, and the second intersection point is the quadrilateral corresponding to the hexagon.

5. The method according to claim 4, characterized in that The masking process is performed on the desk area in the original image to obtain a masked image, including: The original image is input into a first preset model, the desk and chair area in the original image is identified, and the desk and chair area is masked, wherein the first preset model is trained based on sample images and sample annotations, and the sample images include annotated desks and chairs.

6. The method according to any one of claims 1 to 5, characterized in that: Before obtaining the first coordinate information of the target object in the target image, the method further includes: The target object is identified based on the body features of the target object.

7. The method according to claim 6, characterized in that The step of identifying the target object based on the human body features of the target object comprises: Inputting the human body picture of the target object into a second preset model to obtain a first human body feature vector, wherein the second preset model is obtained based on cross-domain data pre-training; Adding the first human body feature vector to a human body feature list; Performing human body detection on each video frame, extracting features of the human body in the first posture, and obtaining a second human body feature vector; The target object is identified based on the similarity value between the second human feature vector and each first human feature vector in the human feature list.

8. The method according to claim 7, characterized in that The identifying the target object based on the similarity value between the second human feature vector and each first human feature vector in the human feature list includes: A similarity value between the second human feature vector and each first human feature vector is obtained, and a maximum value among the similarity values ​​is determined to be greater than a first preset threshold value, and then the human body corresponding to the second human feature vector is determined to be the target object.

9. The method according to claim 8, characterized in that Also includes: If the maximum value of the similarity values ​​is greater than a second preset threshold, the second human feature vector is added to the human feature list, and the second preset threshold is greater than the first preset threshold.

10. The method according to claim 9, characterized in that Also includes: If the number of first human feature vectors in the human feature list is greater than a preset threshold, one or more of the first human feature vectors are deleted.

11. The method according to claim 7, characterized in that Before identifying the target object based on the human features of the target object, the method further includes: Based on face detection, identify the face of the target object in the video frame and obtain a human body image of the target object; or, It is determined that among multiple human bodies in a first posture in the video, a human body in the first posture for the longest time is a target object, and a human body image of the target object is obtained.

12. A moving trajectory tracking device for a target object, characterized in that: include: An acquisition module, used to acquire first coordinate information of a target object in a target image, wherein the target image is an image of a target space acquired by a target camera; A processing module, configured to obtain second coordinate information of the target object in the target space based on the first coordinate information and a perspective transformation matrix; A tracking module is used to determine the moving trajectory of the target object based on the second coordinate information.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for tracking the movement trajectory of the target object as described in any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for tracking the movement trajectory of the target object as described in any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that When the program product is executed by a processor of a vehicle or a cloud server, the steps of the method for tracking the movement trajectory of a target object as described in any one of claims 1 to 11 are implemented.

Images