Region tracking method and device, electronic equipment and storage medium
By extracting and matching image feature points while the camera is moved and determining the location of the target area is solved, the problem of the area when the camera is moved is not possible, and accurate tracking of the target area is achieved.
Patent Information
- Application Number
- CN202310638117.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-31
AI Technical Summary
The prior art cannot effectively track the area marked in the image captured by the camera when the camera is moved, especially when there are pedestrians or cargo moving in the target area. It is impossible to ignore the dynamic movement of the object and only capture the movement of the area.
By acquiring the reference frame image and the frame image to be detected successively by the shooting device, the feature map is extracted and the feature point matching is determined, and the position of the target area in the frame image to be detected is determined based on the feature point position around the calibration point, so as to track the target area .
When the camera moves or pedestrians and cargoes move in the target area, the location of the target area can be accurately tracked to avoid prediction inaccuracy caused by factors such as occlusion, and improve the accuracy of area tracking.
Smart Images

Figure CN119068017A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a region tracking method, device, electronic device and storage medium. Background Art
[0002] The area tracking task is to circle an area in the image captured by a camera, for example, and execute certain applications in the area, such as counting the number of people in the area. These applications will pre-calibrate a polygonal area in the image captured by the camera. The camera may move for various reasons, causing the position of the calibrated area in the image captured by the camera to change, and recalibration is required.
[0003] In the related technology, a similar task - object tracking task is mainly used instead. The object tracking task first uses the model to detect the specific object and extract features, and then detects and compares the similarity of the extracted features between multiple frames to make a consistency judgment. However, the object tracking task tracks the object, while the area tracking task tracks the specific calibrated area. In real scenes, the area is not a static object, and there will be pedestrian movement, cargo movement, etc. When the camera moves, the object tracking task cannot ignore the dynamic movement of the object and only capture the movement of the calibrated area, that is, it cannot achieve area tracking. Summary of the invention
[0004] The present application proposes a region tracking method, device, electronic device and storage medium.
[0005] The first aspect of the present application proposes a region tracking method, including: acquiring a reference frame image and a frame image to be detected taken successively by a shooting device; performing feature map extraction on the reference frame image and the frame image to be detected respectively to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected; performing feature point matching on reference frame feature points in the reference frame feature map and frame feature points to be detected in the frame feature map to be detected; determining the positions of frame calibration points to be detected corresponding to the reference frame calibration points in the frame feature map according to the positions of reference frame calibration points of a reference frame target area calibrated in the reference frame feature map, the positions of a plurality of reference frame feature points within a set range around the reference frame calibration points, and the positions of a plurality of frame feature points to be detected that match the plurality of reference frame feature points in the frame feature map to be detected; and calibrating the frame target area to be detected in the frame image to be detected according to the positions of the frame calibration points to be detected.
[0006] The area tracking method of the embodiment of the present application extracts feature maps of the base frame image and the frame image to be detected successively captured by the shooting device, respectively, to obtain corresponding base frame feature maps and frame feature maps to be detected, performs feature point matching on base frame feature points in the base frame feature map and frame feature points to be detected in the frame feature map to be detected, determines the positions of the calibration points of the frame to be detected in the frame feature map corresponding to the base frame calibration points according to the positions of the base frame calibration points of the base frame target area calibrated in the base frame feature map, the positions of multiple base frame feature points within a set range around the base frame calibration points, and the positions of multiple frame feature points to be detected in the frame feature map to be detected that match the multiple base frame feature points, and calibrates the frame target area to be detected in the frame image to be detected according to the positions of the calibration points of the frame to be detected. The embodiment of the present application extracts feature points of the reference frame image and the frame image to be detected and matches them, and determines the position of the calibration points of the frame to be detected in the frame image to be detected according to the position of the reference frame calibration points in the reference frame image and the positions of multiple matching feature points around the calibration points of the target area in the two frames, and then determines the position of the target area in the frame image to be detected. When the shooting device, such as a camera, moves, the target area can be tracked even if pedestrians or goods move in the target area.
[0007] A second aspect of the present application provides a region tracking device, comprising: an acquisition module, configured to acquire a reference frame image and a frame image to be detected taken successively by a shooting device; an extraction module, configured to perform feature map extraction on the reference frame image and the frame image to be detected, respectively, to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected; a matching module, configured to perform feature point matching on reference frame feature points in the reference frame feature map and frame feature points to be detected in the frame feature map to be detected; a determination module, configured to determine the position of a frame to be detected calibration point in the frame feature map corresponding to the reference frame calibration point in the reference frame target area calibrated in the reference frame feature map, the positions of a plurality of reference frame feature points within a set range around the reference frame calibration point, and the positions of a plurality of frame feature points to be detected that match the plurality of reference frame feature points in the frame feature map to be detected; and a calibration module, configured to calibrate the frame target area to be detected in the frame image to be detected according to the position of the frame calibration point.
[0008] The area tracking device of the embodiment of the present application extracts feature maps of the base frame image and the frame image to be detected successively taken by the shooting device, respectively, to obtain corresponding base frame feature maps and frame feature maps to be detected, performs feature point matching on base frame feature points in the base frame feature map and frame feature points to be detected in the frame feature map to be detected, determines the positions of the calibration points of the frame to be detected in the frame feature map to be detected that correspond to the base frame calibration points according to the positions of the base frame calibration points of the base frame target area calibrated in the base frame feature map, the positions of multiple base frame feature points within a set range around the base frame calibration points, and the positions of multiple frame feature points to be detected in the frame feature map to be detected that match the multiple base frame feature points, and calibrates the frame target area to be detected in the frame image to be detected according to the positions of the calibration points of the frame to be detected. The embodiment of the present application extracts feature points of the reference frame image and the frame image to be detected and matches them, and determines the position of the calibration points of the frame to be detected in the frame image to be detected according to the position of the reference frame calibration points in the reference frame image and the positions of multiple matching feature points around the calibration points of the target area in the two frames, and then determines the position of the target area in the frame image to be detected. When the shooting device, such as a camera, moves, the target area can be tracked even if pedestrians or goods move in the target area.
[0009] The third aspect embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the area tracking method as described in the above-mentioned first aspect embodiment.
[0010] The fourth aspect of the present application provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the area tracking method as described in the first aspect of the present application.
[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0013] Figure 1 A schematic diagram of a flow chart of a region tracking method provided in one embodiment of the present application;
[0014] Figure 2 is a schematic diagram of a reference frame image;
[0015] Figure 3is a schematic diagram of a frame image to be detected;
[0016] Figure 4 is a schematic diagram of a target area calibrated in a reference frame image;
[0017] Figure 5 It is a schematic diagram of the feature point matching graph;
[0018] Figure 6 A schematic diagram of setting a range around a reference frame calibration point;
[0019] Figure 7 is a schematic diagram of a target area marked in a frame image to be detected;
[0020] Figure 8 A schematic diagram of a flow chart of a region tracking method provided by another embodiment of the present application;
[0021] Fig. 9 A schematic diagram of the structure of a region tracking device provided in one embodiment of the present application;
[0022] Fig.10 A block diagram of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION
[0023] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0024] Image processing is a technology that uses computers to analyze images to achieve the desired results. It mainly includes image digitization, image enhancement and restoration, image data encoding, image segmentation and image recognition.
[0025] The following describes the area tracking method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.
[0026] Figure 1 A flowchart of a region tracking method provided in one embodiment of the present application.
[0027] like Figure 1 As shown, the area tracking method of the embodiment of the present application may specifically include the following steps:
[0028] S101, obtaining a reference frame image and a frame image to be detected which are successively photographed by a photographing device.
[0029] Specifically, the execution subject of the area tracking method of the embodiment of the present application may be the area tracking device provided in the embodiment of the present application. The area tracking device may be a hardware device with data information processing capabilities and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc.
[0030] In the embodiment of the present application, the shooting device may specifically include an image acquisition device such as a camera. The shooting device shoots an image, which is recorded as a reference frame image. After a period of time, another image is shot, which is recorded as a frame image to be detected. The frame image to be detected may be an image shot after the shooting device moves, or an image shot when the shooting device does not move. The area tracking device obtains the reference frame image and the frame image to be detected that are shot successively by the shooting device. The reference frame image is as follows: Figure 2 As shown, the frame image to be detected is Figure 3 As shown, by comparison Figure 2 and Figure 3 , it can be determined that the shooting device has moved.
[0031] The target area is marked in the reference frame image, which is recorded as the reference frame target area. Figure 4 As shown. The purpose of the embodiment of the present application is to recalibrate the target area in the frame image to be detected, that is, the target area of the frame to be detected. If the frame image to be detected is taken after the shooting device moves, the position of the target area of the frame to be detected in the frame image to be detected is also moved compared with the position of the target area of the reference frame in the reference frame image. If the frame image to be detected is taken when the shooting device does not move, the position of the target area of the frame to be detected in the frame image to be detected is also not moved compared with the position of the target area of the reference frame in the reference frame image.
[0032] S102, extracting feature maps of the reference frame image and the frame image to be detected respectively, to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected.
[0033] Specifically, the region tracking device extracts a feature map from the reference frame image obtained in step S101 to obtain a feature map corresponding to the reference frame image, namely, a reference frame feature map. The region tracking device extracts a feature map from the frame image to be detected obtained in step S101 to obtain a feature map corresponding to the frame image to be detected, namely, a feature map of the frame to be detected. The reference frame feature map includes feature points extracted from the reference frame image, namely, reference frame feature points, and the feature map of the frame to be detected includes feature points extracted from the frame image to be detected, namely, feature points of the frame to be detected.
[0034] S103, performing feature point matching on the reference frame feature points in the reference frame feature map and the feature points on the frame to be detected in the frame to be detected feature map.
[0035] Specifically, the region tracking device performs feature point matching on the reference frame feature points in the reference frame feature map obtained in step S102 and the feature points of the frame to be detected in the feature map of the frame to be detected, and obtains a feature point matching map such as Figure 5 As shown. Feature point matching is to find the same feature point in two frames of images. Generally, the similarity of two feature points in two frames of images is compared to determine whether the two feature points match. Each feature point corresponds to a descriptor, which is generally represented by a vector. The similarity of the two feature points can be obtained by calculating the similarity of the two vectors.
[0036] S104, determining the positions of the calibration points of the frame to be detected corresponding to the reference frame calibration points in the feature map of the frame to be detected according to the positions of the reference frame calibration points of the reference frame target area calibrated in the reference frame feature map, the positions of multiple reference frame feature points within a set range around the reference frame calibration points, and the positions of multiple feature points of the frame to be detected that match the multiple reference frame feature points in the feature map of the frame to be detected.
[0037] Specifically, the calibration points of the target area are the corner points of the target area, the calibration points of the reference frame are the corner points of the target area of the reference frame, and the calibration points of the frame to be detected are the corner points of the target area of the frame to be detected. The set range is a pre-set range, which can be a circular area with the calibration point of the reference frame as the center and the set pixel distance as the radius, such as Figure 6 As shown. In actual application, the setting range can be adjusted according to the number of reference frame feature points within the setting range. For example, when the number of reference frame feature points within the setting range is 0-1, the setting range can be appropriately expanded. The set pixel distance can be specifically set to 3-5 pixel distances. Taking the 3 pixel distance as an example, assuming that the reference frame calibration point is located in the 5th row and the 5th column, the reference frame feature point in the 1st row and the 5th column does not belong to the feature points within the surrounding setting range, and the reference frame feature point in the 3rd row and the 5th column belongs to the feature points within the surrounding setting range.
[0038] The area tracking device obtains the position of the reference frame calibration point of the reference frame target area that has been calibrated in the reference frame feature map, obtains the positions of multiple reference frame feature points within a set range around the reference frame calibration point, obtains multiple feature points of the frame to be detected that match the above-mentioned multiple reference frame feature points in the feature map of the frame to be detected and their positions based on the feature matching map obtained in step S103, and determines the position of the calibration point of the frame to be detected in the feature map of the frame to be detected corresponding to the reference frame calibration point according to the above-mentioned positions obtained.
[0039] S105, calibrating a target area of the frame to be detected in the image of the frame to be detected according to the positions of the calibration points of the frame to be detected.
[0040] Specifically, the region tracking device recalibrates the target region in the image of the frame to be detected according to the position of the calibration point of the frame to be detected determined in step S104, that is, recalibrates the target region of the frame to be detected, such as Figure 7 shown.
[0041] In summary, the area tracking method of the embodiment of the present application extracts feature maps of the base frame image and the frame image to be detected successively captured by the shooting device, respectively, to obtain corresponding base frame feature maps and feature maps of the frame to be detected, performs feature point matching on the base frame feature points in the base frame feature map and the frame feature points to be detected in the frame feature map to be detected, and determines the positions of the calibration points of the frame to be detected corresponding to the base frame calibration points in the frame feature map to be detected, according to the positions of the base frame calibration points of the base frame target area calibrated in the base frame feature map, the positions of multiple base frame feature points within a set range around the base frame calibration points, and the positions of multiple frame feature points to be detected that match the multiple base frame feature points in the frame feature map to be detected, and calibrates the frame target area to be detected in the frame image to be detected according to the positions of the calibration points of the frame to be detected. The embodiment of the present application extracts and matches the feature points of the reference frame image and the frame image to be detected, and determines the position of the calibration points of the frame to be detected in the frame image to be detected based on the position of the reference frame calibration points in the reference frame image and the positions of multiple matching feature points around the calibration points of the target area in the two frames, and then determines the position of the target area in the frame image to be detected. When the shooting device, such as a camera, moves, the target area can be tracked even if pedestrians or goods move in the target area. The position of the calibration points of the frame to be detected is predicted based on the positions of multiple reference frame feature points within a set range around the reference frame calibration points, and the method of comprehensive prediction of multiple points is used to avoid the risk of inaccurate prediction due to factors such as occlusion.
[0042] Figure 8 FIG. 1 is a flow chart of a region tracking method provided by another embodiment of the present application. Figure 8 As shown in the above Figure 1 Based on the illustrated embodiment, the area tracking method of the embodiment of the present application may specifically include the following steps:
[0043] S801, obtaining a reference frame image and a frame image to be detected which are successively photographed by a photographing device.
[0044] Specifically, step S801 in this embodiment is the same as step S101 in the above embodiment, and will not be described in detail here.
[0045] S802, performing image preprocessing on the reference frame image and the frame image to be detected respectively.
[0046] Specifically, before extracting feature maps from the reference frame image and the frame image to be detected, the reference frame image and the frame image to be detected may be preprocessed, and then feature maps may be extracted from the preprocessed images to improve the accuracy of feature map extraction. Image preprocessing may specifically include but is not limited to grayscale, normalization, and other processing.
[0047] In the above embodiment, step S102 of "extracting feature maps of the reference frame image and the frame image to be detected respectively to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected" may specifically include the following step S803.
[0048] S803, using a deep neural network to extract feature maps of the reference frame image and the frame image to be detected, respectively, to obtain a reference frame feature map and a frame feature map to be detected.
[0049] Specifically, non-deep learning methods, such as optical flow-based solutions, can be used to extract feature maps, or deep learning methods, such as deep neural networks, can be used to extract feature maps. Using deep learning methods to extract feature maps can improve the quality of feature map extraction and save a lot of computing resources.
[0050] In the above embodiment, step S103 of “matching feature points of the reference frame feature points in the reference frame feature map with feature points of the frame to be detected in the frame to be detected feature map” may specifically include the following steps S804-S805.
[0051] S804, calculating the Euclidean distance between the feature points of the reference frame and the feature points of the frame to be detected.
[0052] Specifically, each feature point corresponds to a descriptor, which is generally represented by a vector. The similarity of the two vectors, that is, the similarity of the two feature points, can be obtained by calculating the Euclidean distance between the two vectors.
[0053] Euclidean distance is the "ordinary" (i.e. straight-line) distance between two points represented by vectors in Euclidean space.
[0054] S805 , matching feature points of the reference frame and feature points of the frame to be detected according to the Euclidean distance.
[0055] Specifically, if the Euclidean distance between the reference frame feature point calculated in step S804 and the feature point of the frame to be detected is less than a preset Euclidean distance threshold, it is determined that the reference frame feature point matches the feature point of the frame to be detected. If the Euclidean distance between the reference frame feature point calculated in step S804 and the feature point of the frame to be detected is equal to or greater than a preset Euclidean distance threshold, it is determined that the reference frame feature point does not match the feature point of the frame to be detected.
[0056] Step S104 in the above embodiment, "determine the positions of the calibration points of the frame to be detected corresponding to the reference frame calibration points in the feature map of the frame to be detected according to the positions of the reference frame calibration points of the reference frame target area calibrated in the reference frame feature map, the positions of multiple reference frame feature points within a set range around the reference frame calibration points, and the positions of multiple feature points of the frame to be detected that match the multiple reference frame feature points in the feature map of the frame to be detected", may specifically include the following steps S806-S809.
[0057] S806, calculating the position of the center point of the plurality of reference frame feature points according to the positions of the plurality of reference frame feature points.
[0058] S807, calculating a position vector from the reference frame calibration point to the center point of the reference frame feature point according to the position of the reference frame calibration point and the position of the center point of the reference frame feature point.
[0059] Specifically, assuming that the position of the reference frame calibration point A is (x1, y1), and the position of the center point B of the reference frame feature point is (x2, y2), then the position vector AB from the reference frame calibration point A to the center point B of the reference frame feature point is calculated to be (x2-x1, y2-y1).
[0060] S808, calculating the position of the center point of the plurality of frame feature points to be detected according to the positions of the plurality of frame feature points to be detected.
[0061] S809, determining the position of the calibration point of the frame to be detected according to the position vector and the position of the center point of the feature point of the frame to be detected.
[0062] Specifically, assuming that the position of the center point B' of the feature point of the frame to be detected is (x2', y2'), the position of the calibration point A' of the frame to be detected is determined to be (x2', y2')-(x2-x1, y2-y1)=(x2'-x2+x1, y2'-y2+y1).
[0063] S810, calibrating a target area of the frame to be detected in the image of the frame to be detected according to the positions of the calibration points of the frame to be detected.
[0064] Specifically, step S810 in this embodiment is the same as step S105 in the above embodiment, and will not be described in detail here.
[0065] In summary, the area tracking method of the embodiment of the present application extracts feature maps of the base frame image and the frame image to be detected successively captured by the shooting device, respectively, to obtain corresponding base frame feature maps and feature maps of the frame to be detected, performs feature point matching on the base frame feature points in the base frame feature map and the frame feature points to be detected in the frame feature map to be detected, and determines the positions of the calibration points of the frame to be detected corresponding to the base frame calibration points in the frame feature map to be detected, according to the positions of the base frame calibration points of the base frame target area calibrated in the base frame feature map, the positions of multiple base frame feature points within a set range around the base frame calibration points, and the positions of multiple frame feature points to be detected that match the multiple base frame feature points in the frame feature map to be detected, and calibrates the frame target area to be detected in the frame image to be detected according to the positions of the calibration points of the frame to be detected. The embodiment of the present application extracts and matches the feature points of the reference frame image and the frame image to be detected, and determines the position of the frame calibration point to be detected in the frame image to be detected according to the position of the reference frame calibration point in the reference frame image and the position of multiple matching feature points around the calibration points of the target area in the two frames, and then determines the position of the target area in the frame image to be detected. When the shooting device, such as the camera, moves, even if pedestrians and goods move in the target area, the target area can be tracked. The position of the frame calibration point to be detected is predicted based on the position of multiple reference frame feature points within a set range around the reference frame calibration point. The risk of inaccurate prediction due to factors such as occlusion can be avoided by using a multi-point comprehensive prediction method. By calculating the position vector of the reference frame calibration point to the center point of multiple reference frame feature points, the relative position relationship between the reference frame calibration point and the surrounding reference frame feature points can be quickly and accurately determined, and then the position of the frame calibration point to be detected can be quickly and accurately determined based on the relative position relationship and the corresponding position of the center point of multiple frame feature points to be detected. The Euclidean distance is used for feature point matching to improve the matching accuracy. Preprocessing the image and using neural network to extract feature maps further improve the accuracy of feature map extraction.
[0066] In order to implement the above embodiment, the embodiment of the present application also proposes a region tracking device. Fig. 9 FIG. 1 is a schematic diagram of the structure of a region tracking device according to an embodiment of the present application. Fig. 9 As shown, the region tracking device 900 of the embodiment of the present application may specifically include: an acquisition module 901, an extraction module 902, a matching module 903, a determination module 904 and a calibration module 905. Among them:
[0067] The acquisition module 901 is configured to acquire the reference frame image and the frame image to be detected which are successively photographed by the photographing device.
[0068] The extraction module 902 is configured to extract feature maps of the reference frame image and the frame image to be detected respectively, to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected.
[0069] The matching module 903 is configured to perform feature point matching on the reference frame feature points in the reference frame feature map and the feature points of the frame to be detected in the frame to be detected feature map.
[0070] The determination module 904 is configured to determine the positions of the calibration points of the frame to be detected corresponding to the reference frame calibration points in the feature map of the frame to be detected based on the positions of the reference frame calibration points of the reference frame target area calibrated in the reference frame feature map, the positions of multiple reference frame feature points within a set range around the reference frame calibration points, and the positions of multiple feature points of the frame to be detected that match the multiple reference frame feature points in the feature map of the frame to be detected.
[0071] The calibration module 905 is configured to calibrate the target area of the frame to be detected in the image of the frame to be detected according to the positions of the calibration points of the frame to be detected.
[0072] In one embodiment of the present application, the determination module 904 is further configured to: calculate the position of the center point of multiple reference frame feature points based on the positions of multiple reference frame feature points; calculate the position vector from the reference frame calibration point to the center point of the reference frame feature point based on the position of the reference frame calibration point and the position of the center point of the reference frame feature point; calculate the position of the center point of multiple frame feature points to be detected based on the positions of multiple frame feature points to be detected; determine the position of the frame calibration point to be detected based on the position vector and the position of the center point of the frame feature point to be detected.
[0073] In one embodiment of the present application, the set range is a circular area with the reference frame calibration point as the center and the set pixel distance as the radius.
[0074] In one embodiment of the present application, the pixel distance is set to 3-5 pixels.
[0075] In one embodiment of the present application, the matching module 903 is further configured to: calculate the Euclidean distance between the feature points of the reference frame and the feature points of the frame to be detected; and perform feature point matching on the feature points of the reference frame and the feature points of the frame to be detected according to the Euclidean distance.
[0076] In one embodiment of the present application, the matching module 903 is further configured to: if the Euclidean distance is less than a preset Euclidean distance threshold, determine that the feature point of the reference frame matches the feature point of the frame to be detected.
[0077] In one embodiment of the present application, the extraction module 902 is further configured to: perform image preprocessing on the reference frame image and the frame image to be detected before extracting feature maps on the reference frame image and the frame image to be detected respectively.
[0078] In one embodiment of the present application, the extraction module 902 is further configured to: use a deep neural network to extract feature maps of the reference frame image and the frame image to be detected, respectively, to obtain a reference frame feature map and a frame feature map to be detected.
[0079] It should be noted that the above explanation of the embodiment of the area tracking method is also applicable to the area tracking device of the embodiment of the present application, and the specific process will not be repeated here.
[0080] In summary, the area tracking device of the embodiment of the present application extracts feature maps of the base frame image and the frame image to be detected successively captured by the shooting device, respectively, to obtain corresponding base frame feature maps and feature maps of the frame to be detected, performs feature point matching on the base frame feature points in the base frame feature map and the frame feature points to be detected in the frame feature map to be detected, determines the positions of the calibration points of the frame to be detected corresponding to the base frame calibration points in the frame feature map to be detected, and calibrates the frame target area to be detected in the frame image to be detected according to the positions of the base frame calibration points of the base frame target area calibrated in the base frame feature map, the positions of multiple base frame feature points within a set range around the base frame calibration points, and the positions of multiple frame feature points to be detected that match the multiple base frame feature points in the frame feature map to be detected. The embodiment of the present application extracts and matches the feature points of the reference frame image and the frame image to be detected, and determines the position of the frame calibration point to be detected in the frame image to be detected according to the position of the reference frame calibration point in the reference frame image and the position of multiple matching feature points around the calibration points of the target area in the two frames, and then determines the position of the target area in the frame image to be detected. When the shooting device, such as the camera, moves, even if pedestrians and goods move in the target area, the target area can be tracked. The position of the frame calibration point to be detected is predicted based on the position of multiple reference frame feature points within a set range around the reference frame calibration point. The risk of inaccurate prediction due to factors such as occlusion can be avoided by using a multi-point comprehensive prediction method. By calculating the position vector of the reference frame calibration point to the center point of multiple reference frame feature points, the relative position relationship between the reference frame calibration point and the surrounding reference frame feature points can be quickly and accurately determined, and then the position of the frame calibration point to be detected can be quickly and accurately determined based on the relative position relationship and the corresponding position of the center point of multiple frame feature points to be detected. The Euclidean distance is used for feature point matching to improve the matching accuracy. Preprocessing the image and using neural network to extract feature maps further improve the accuracy of feature map extraction.
[0081] like Fig.10, is a block diagram of an electronic device according to the area tracking method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as intelligent voice interaction devices, personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0082] like Fig.10 As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor 1001 can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 1001 is taken as an example.
[0083] The memory 1002 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the region tracking method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the region tracking method provided in the present application.
[0084] The memory 1002 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the area tracking method in the embodiment of the present application (for example, the attached Fig. 9 The processor 1001 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1002, that is, the area tracking method in the above method embodiment is implemented.
[0085] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the electronic device of the regional tracking method, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the electronic device of the regional tracking method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0086] The electronic device of the area tracking method may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.
[0087] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the area tracking method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0088] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0089] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0090] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0091] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0092] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short).
[0093] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0094] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0095] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A region tracking method, characterized in that: include: Acquire a reference frame image and a frame image to be detected that are successively captured by a shooting device; Extracting feature maps of the reference frame image and the frame image to be detected respectively to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected; Performing feature point matching on the reference frame feature points in the reference frame feature map and the frame feature points to be detected in the frame feature map to be detected; Determine the position of the frame calibration point to be detected in the frame feature map to be detected corresponding to the reference frame calibration point according to the position of the reference frame calibration point of the reference frame target area calibrated in the reference frame feature map, the positions of a plurality of the reference frame feature points within a set range around the reference frame calibration point, and the positions of a plurality of the frame feature points to be detected in the frame feature map to be detected that match the plurality of the reference frame feature points; The target area of the frame to be detected is calibrated in the image of the frame to be detected according to the positions of the calibration points of the frame to be detected.
2. The method according to claim 1, characterized in that The method of determining the position of the frame calibration point to be detected corresponding to the reference frame calibration point in the frame feature map to be detected according to the position of the reference frame calibration point of the reference frame target area calibrated in the reference frame feature map, the positions of a plurality of the reference frame feature points within a set range around the reference frame calibration point, and the positions of a plurality of the frame feature points to be detected in the frame feature map to be detected that match the plurality of the reference frame feature points, comprises: Calculate the position of the center point of the plurality of reference frame feature points according to the positions of the plurality of reference frame feature points; Calculating a position vector from the reference frame calibration point to the center point of the reference frame feature point according to the position of the reference frame calibration point and the position of the center point of the reference frame feature point; Calculate the position of the center point of the plurality of frame feature points to be detected according to the positions of the plurality of frame feature points to be detected; The position of the calibration point of the frame to be detected is determined according to the position vector and the position of the center point of the feature point of the frame to be detected.
3. The method according to claim 1, characterized in that: The set range is a circular area with the reference frame calibration point as the center and the set pixel distance as the radius.
4. The method according to claim 3, characterized in that The set pixel distance is 3-5 pixels.
5. The method according to claim 1, characterized in that The matching of feature points of the reference frame feature points in the reference frame feature map with feature points of the frame to be detected in the frame to be detected feature map comprises: Calculating the Euclidean distance between the reference frame feature point and the to-be-detected frame feature point; Feature point matching is performed on the reference frame feature points and the feature points of the frame to be detected according to the Euclidean distance.
6. The method according to claim 5, characterized in that The matching of the feature points of the reference frame and the feature points of the frame to be detected according to the Euclidean distance includes: If the Euclidean distance is less than a preset Euclidean distance threshold, it is determined that the reference frame feature point matches the feature point of the frame to be detected.
7. The method according to claim 1, characterized in that Before extracting the feature maps of the reference frame image and the frame image to be detected respectively, the method further includes: Image preprocessing is performed on the reference frame image and the frame image to be detected respectively.
8. The method according to claim 1, characterized in that: The extracting feature maps of the reference frame image and the frame image to be detected respectively to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected comprises: A deep neural network is used to extract feature maps of the reference frame image and the frame image to be detected, respectively, to obtain the reference frame feature map and the frame feature map to be detected.
9. A region tracking device, characterized in that: include: An acquisition module is configured to acquire a reference frame image and a frame image to be detected which are successively captured by a shooting device; An extraction module is configured to extract feature maps of the reference frame image and the frame image to be detected respectively, to obtain a reference frame feature map corresponding to the reference frame image and a frame feature map to be detected corresponding to the frame image to be detected; A matching module is configured to perform feature point matching on the reference frame feature points in the reference frame feature map and the frame feature points to be detected in the frame feature map to be detected; A determination module is configured to determine the position of the frame calibration point to be detected in the frame feature map to be detected corresponding to the reference frame calibration point according to the position of the reference frame calibration point of the reference frame target area calibrated in the reference frame feature map, the positions of a plurality of the reference frame feature points within a set range around the reference frame calibration point, and the positions of a plurality of the frame feature points to be detected in the frame feature map to be detected that match the plurality of the reference frame feature points; The calibration module is configured to calibrate the target area of the frame to be detected in the image of the frame to be detected according to the position of the calibration point of the frame to be detected.
10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
11. A computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Reverse-structure-matched-tracking-based accurate automatic calibration method for sequence image target point
CN107945209A
Calibration method, device and system and storage device
CN110766758A
Pose estimation method based on feature point combination matching of calibration area
CN111553395A
Target detection method and device, related equipment and computer readable storage medium
CN113505760A
Plane tracking method and system based on feature points
CN113689466A
