A method and device for finding items based on video playback

Abstract

This invention relates to the field of image processing technology, and more particularly to a method and apparatus for item retrieval based on video backtracking. The method includes: determining the item characteristics, search area, and search time period of the item to be found based on a received item retrieval request; wherein the item characteristics include at least one of category features, appearance features, and image features; determining the video to be found based on the search time period and the search area; determining keyframe images of the video to be found based on the motion vectors of feature points in each frame of the video; performing image recognition on the keyframe images based on the item characteristics to determine the bounding box coordinates and timestamp of the item to be found in each keyframe image; and determining the search area of ​​the item to be found based on the timestamp and the bounding box coordinates. Compared with existing technologies, this invention has advantages such as more accurate filtering of keyframe images and improved efficiency and accuracy in item retrieval.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and device for finding objects based on video backtracking. Background Technology

[0002] Losing items is a common problem in daily life. Whether in public places such as shopping malls, airports, and restaurants, or at home, once an item is lost, it often requires manual identification by the naked eye. However, reviewing a large amount of surveillance footage is extremely inefficient, prone to omissions, and only suitable for short periods and small-scale scenarios. This process is not only tedious and time-consuming, but also prone to missing crucial information due to visual fatigue.

[0003] Object-detection-based methods, such as those using YOLO and Faster R-CNN, identify targets in videos that match the features of the item being searched, extracting the corresponding frames to determine its location. Combined with ReID (Person Re-identification) technology, they track individuals associated with the item, indirectly locating the object. However, these methods share common limitations: they only identify frames containing the item, without analyzing its dynamic trajectory; they lack inter-frame motion information, resulting in redundant keyframes; and they fail to narrow the search scope by incorporating the "search area / time period," leading to significant computational overhead.

[0004] A search revealed Chinese invention patent application publication number CN115937310A, which discloses a traceable intelligent object-finding system based on YOLO and its control method. The system includes an intelligent camera, a recognition subsystem, an object-finding subsystem, and a deployment module. The intelligent camera is used to acquire video signals. The recognition subsystem filters video signals based on target information using a YOLO network to obtain target video signals and identify targets within them. The object-finding subsystem extracts and processes keyframes of the target in the video signal to determine the target's location. The deployment module accelerates the deployment of the object-finding subsystem within the intelligent camera. However, this existing patent application has the following problems: First, it relies solely on the YOLO network for target filtering without analyzing motion information between video frames, making it difficult to accurately capture the dynamic movement of objects. Second, the keyframe extraction lacks filtering logic based on feature point motion vectors, easily including a large number of invalid frames, leading to computational redundancy and low object-finding efficiency. Third, it does not combine the limiting conditions of the search area and search time period to narrow the search scope, still resulting in excessive video data processing volume.

[0005] How to automatically and quickly retrieve lost items and improve the accuracy and efficiency of item retrieval has become a technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for finding items based on video backtracking.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] According to one aspect of the present invention, a method for finding items based on video playback is provided, comprising:

[0009] Based on the received item search request, the item characteristics, search area, and search time period of the item to be searched are determined; wherein the item characteristics include at least one of category characteristics, appearance characteristics, and image characteristics;

[0010] Based on the time period to be searched and the area to be searched, the video to be searched is determined;

[0011] Based on the motion vectors of feature points in each frame of the video to be searched, the keyframe images of the video to be searched are determined. Specifically, this includes: traversing the input video to be searched; for each frame image to be detected in the video to be searched, determining the motion vector features of all feature points in the frame image to be detected and the next frame image, wherein the motion vector features include the average motion amplitude and the proportion of feature points whose motion amplitude is greater than a preset threshold; determining the historical motion vector features of the frame image to be detected within a preset historical time period, and using the historical motion vector features as a measurement threshold for the frame image to be detected; when the motion vector features of the frame image to be detected are higher than the measurement threshold, the frame image to be detected is used as the keyframe image of the video to be searched.

[0012] Based on the features of the item, image recognition is performed on the keyframe images to determine the bounding box coordinates and timestamp of the item to be found in each keyframe image;

[0013] Based on the timestamp and the bounding box coordinates, the search area for the item to be found is determined.

[0014] As a preferred technical solution, the method further includes: determining feature points in each frame of the video to be searched before determining the keyframe images of the video to be searched, specifically:

[0015] Iterate through each frame of the input video to be searched. For each pixel to be detected in each frame, determine the first gray intensity of the pixel to be detected and the second gray intensity set of other pixels within a preset range of the pixel to be detected.

[0016] If all second grayscale intensities within the second grayscale intensity set are higher than the sum of the first grayscale intensity and a preset threshold, or are all lower than the difference between the first grayscale intensity and the preset threshold, then the pixel to be detected is taken as the feature point; and / or,

[0017] Create a two-dimensional uniform grid on the video frame image;

[0018] The pixel positions of all intersections or center points of the grid are used as the feature points.

[0019] As a preferred technical solution, the process of determining the key frame images of the video to be searched further includes: receiving the minimum key frame interval set by the user, and determining the frame interval between each adjacent key frame image.

[0020] As a preferred technical solution, after determining the frame interval between each adjacent keyframe image, the method further includes: in response to the frame interval between adjacent keyframe images being less than the minimum keyframe interval, performing image recognition on two adjacent keyframe images based on the item features to determine the image confidence of the two keyframe images;

[0021] Delete keyframe images whose image confidence is lower than the confidence threshold until the frame interval between any two adjacent keyframe images is greater than the minimum keyframe interval.

[0022] As a preferred technical solution, the video to be searched is determined based on the time period to be searched and the region to be searched, specifically including:

[0023] In response to the fact that the number of times the item characteristics of the item to be found appear within a preset time period is higher than the first time threshold, the historical retrieval location of the item to be found is obtained;

[0024] When the number of times the historical retrieval location is located in the area to be searched is higher than the second threshold, the video to be searched is determined based on the historical retrieval location, the time period to be searched, and the area to be searched.

[0025] When the number of times the historical retrieval location is located in the area to be searched is not higher than the second number threshold, the video to be searched is determined based on the time period to be searched and the area to be searched.

[0026] As a preferred technical solution, determining the search area for the item to be found specifically includes:

[0027] Based on continuous timestamps and bounding box coordinates, the motion trajectory of the object to be found in the video to be found is constructed;

[0028] Extrapolate the motion trajectory to predict the possible direction of movement and destination area of ​​the object after it leaves the monitoring field of view;

[0029] The possible directions of movement and destination areas are merged with the electronic map of the area to be searched to delineate multiple candidate search areas with unequal probabilities on the electronic map, and the candidate search areas are provided to the user.

[0030] As a preferred technical solution, the method also supports determining the search area for the item across monitoring devices, specifically including:

[0031] Based on the timestamp and the bounding box coordinates, the disappearance time and disappearance area of ​​the item to be found in the video to be found are determined;

[0032] In response to the disappearance area being within the monitoring range of other monitoring devices, based on the disappearance time and the disappearance area, subsequent videos to be searched by other monitoring devices are determined;

[0033] Image recognition is performed based on key frame images of the subsequent video to be searched until the final disappearance area is no longer within the monitoring range, or the search area of ​​the item to be searched is determined.

[0034] Send the final disappearance area or the search area to the user.

[0035] As a preferred technical solution, the process of performing image recognition on the keyframe image based on the features of the item includes:

[0036] In response to the failure to identify the item by image recognition, audio data synchronized with the video being searched is acquired;

[0037] Based on natural language processing technology, keywords related to the item to be found are identified from the audio data;

[0038] Locate the time point when the keyword appears, and extract the video frame at the corresponding time point as a supplementary keyframe;

[0039] Based on the characteristics of the item, image recognition is performed on the supplementary keyframes to locate the item to be found.

[0040] As a preferred technical solution, if a user loses the same item multiple times, historical item search data is used to optimize the current item search process, specifically including:

[0041] The system automatically adjusts the search area and search time period based on historical search data, and uses the item location information in the historical search data to adjust the key frame image screening criteria and image recognition parameters in the current search process.

[0042] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) This invention first determines the characteristics of the item, the area to be searched, and the time period to be searched based on the item search request. Then, it determines the key frame images of the video to be searched. The key frame images are determined based on the motion vector characteristics of the feature points between adjacent frames of the video to be searched (including the average motion amplitude and the proportion of feature points with large motion). The measurement threshold dynamically adapts with the historical motion vector characteristics, which is more in line with the actual motion law than a fixed threshold. Based on the item characteristics, the key frame images are image recognized to identify the bounding box coordinates and timestamp of the item to be searched, and to accurately and efficiently determine the search area of ​​the item to be searched.

[0045] 2) In order to avoid the situation of excessively dense keyframes, the present invention sets a small keyframe interval to ensure that there is a sufficient frame interval between adjacent keyframe images. Then, keyframe images with low image confidence are deleted to perform secondary screening and optimization of keyframe images, so as to efficiently and accurately find items in high-quality keyframe images.

[0046] 3) This invention also supports multimodal data processing including audio for item retrieval, identifying keywords related to the item to be found from the audio data, locating the time point of the keyword appearance, and extracting the video frame at the corresponding time point as a supplementary key frame; performing image recognition on the supplementary key frame based on the item features to find the item to be found. The combination of multimodal data can improve the success rate and accuracy of item retrieval.

[0047] 4) This invention addresses the issue of repeatedly lost items by fully utilizing historical search data to optimize keyframe image screening criteria and image recognition parameters, thereby improving the accuracy and efficiency of item retrieval and providing users with a more convenient and efficient item retrieval experience. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the video-based item search method of the present invention.

[0049] Figure 2 This is a schematic diagram of the process of finding items using historical data in this invention;

[0050] Figure 3 This is a schematic diagram of the structure of the video-based item retrieval device in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] This embodiment relates to a method for finding items based on video playback, such as... Figure 1 ,include:

[0053] S101: In response to receiving a lost item request from a user, based on the lost item request, determine the characteristics of the lost item, the area to be searched, and the time period to be searched.

[0054] First, when a user discovers an item is lost, they send a lost item request to the server. Upon receiving the request, the server interacts with the user to determine the item's characteristics, the search area, and the search period. Item characteristics can include category characteristics, such as whether it's a phone, wallet, or keys; appearance characteristics, such as color, shape, and size; and image characteristics, i.e., images of the item provided by the user. The search area refers to the range of locations the user believes the item might have been lost, such as a specific floor in a shopping mall or a certain area in a restaurant. The search period is the time period the user remembers the item might have been lost during.

[0055] S102: Determine the video to be searched based on the time period and the area to be searched.

[0056] After determining the characteristics of the item to be searched, the search area, and the search time period, the server will further filter relevant surveillance videos based on this information; these are the videos to be searched. The purpose of this step is to narrow down the scope of video processing and improve the efficiency and accuracy of subsequent processing. When determining the videos to be searched, the server can query the database for surveillance video records that match the search area and time period, and then filter out videos that meet the criteria.

[0057] S103: Determine the keyframe images of the video to be searched based on the motion vectors of feature points within each video frame.

[0058] After acquiring the video to be searched, the server analyzes each frame, calculating the motion vectors of feature points in each frame between adjacent frames. These motion vectors reflect the direction and speed of the feature points' movement in the video. By analyzing these motion vectors, the server can identify keyframe images in the video. Keyframe images are typically those frames containing significant motion changes or scene transitions, playing a crucial role in subsequent image recognition and processing. By filtering keyframe images, the server can significantly reduce the amount of video data that needs to be processed, thereby improving processing speed and efficiency.

[0059] S104: Based on the object features, perform image recognition on the keyframe images to determine the bounding box coordinates and timestamp of the object to be found in each keyframe image.

[0060] After identifying the keyframe images, the server uses advanced image recognition technology to analyze each keyframe image to find targets that match the features of the item being searched for. The image recognition process can be based on deep learning algorithms, improving accuracy and robustness through training with a large amount of sample data. Once the item is found in the keyframe image, the server records the bounding box coordinates and timestamp of the item. The bounding box coordinates identify the item's specific location in the image, while the timestamp records the corresponding time point in that frame.

[0061] S105: Determine the search area for the item based on the timestamp and bounding box coordinates.

[0062] After obtaining the bounding box coordinates and timestamp of the item to be found in the keyframe image, the server can further determine the search area for the item based on this information. The search area refers to the location where the item last appeared within the monitoring range or the area it may have moved to. Specifically, the server combines the changes in the bounding box coordinates of the item in consecutive frames to analyze its movement trajectory, thereby determining the area where the item may appear or its current location.

[0063] In S103, before determining the keyframe images, it is necessary to determine the feature points in each frame. Here, feature points refer to pixels in a video frame that have significant features and can represent the motion information of that frame. These feature points typically have unique grayscale, texture, or color characteristics, enabling them to be accurately tracked and matched between adjacent frames. To determine these feature points, the server provides two methods.

[0064] One approach is to iterate through the input video frames to be detected, and for each pixel to be detected, calculate its first grayscale intensity (for ease of explanation later, we will use...). (This is represented as) the grayscale value of the pixel, and at the same time, the second grayscale intensity of other pixels within a preset range (e.g., a range with a diameter of 3 pixels centered on the pixel to be detected, where there are a total of 8 other pixels) is calculated (for ease of explanation later, this is represented as) the grayscale value of the pixel. (This is represented as a set.) If all second grayscale intensities within the second grayscale intensity set are higher than the first grayscale intensity and a preset threshold (for ease of explanation later, this is used as...), then... The sum of (represented) (i.e., for the 8 other pixels, we have:) ), or all are lower than the difference between the first grayscale intensity and the preset threshold (i.e., for the other 8 pixels, all have: If the pixel to be detected is a feature point, then that pixel is considered a feature point. This method, based on the comparison of pixel grayscale intensity, can effectively filter out pixels with significant features as feature points.

[0065] Another approach is to create a two-dimensional uniform grid on the video frame image, using the pixel positions of all grid intersections or center points as feature points. This method is simple and intuitive, and can evenly distribute feature points, thus covering the entire video frame image. By determining these feature points, the server can further analyze their motion vectors between adjacent frames, providing a foundation for subsequent keyframe image selection and image recognition.

[0066] In step S103, after obtaining the motion vectors of feature points in each video frame, the determination of keyframe images can be achieved by calculating the motion vector features of feature points between adjacent frames. These motion vector features include the average motion amplitude or the proportion of feature points with motion amplitudes greater than a threshold. The threshold can be determined by the historical motion vector features of the frame image to be detected within a preset historical time period (e.g., the first 30 frames), and these historical motion vector features are used as the threshold. When the motion vector features of the frame image to be detected are higher than the threshold, it is used as the keyframe image of the video to be searched. If the motion vector features of the frame image to be detected are not higher than the threshold, the next frame image is analyzed until all frames of the entire video to be searched have been traversed, thus filtering out all keyframe images that meet the conditions. This method, by quantifying the motion changes of feature points, can accurately capture key motion scenes and keyframe images in the video, providing strong support for subsequent image recognition and object search region determination. This method uses the historical motion vector features (first 30 frames) of the frame to be detected as the threshold for the current frame, achieving dynamic adaptation of the threshold. At the same time, it uses the motion changes of quantized feature points to determine the key frame image, making the filtering logic more in line with the actual motion law of the video compared to a fixed threshold.

[0067] After obtaining the keyframe images, since there may be a large number of keyframes (e.g., multiple keyframes during user movement), the minimum keyframe interval set by the user is used to ensure sufficient frame spacing between adjacent keyframe images, avoiding overly dense keyframes. For the marked potential keyframe images, image recognition is further performed based on item features, and image confidence is calculated. Keyframe images with low confidence (below the confidence threshold) are deleted until the minimum keyframe interval requirement is met, thus determining the final keyframe images. When calculating image confidence, various advanced image recognition algorithms can be used to comprehensively evaluate the degree of matching between the image and the item features of the target item. For example, a convolutional neural network model in deep learning can be used. This model, trained on a large number of samples containing features such as the category and appearance of the target item, can perform detailed analysis of keyframe images and output a confidence score reflecting the degree of matching. If a keyframe image has a low confidence score, it indicates that the target item may not exist in the image or the recognition result is unreliable; in this case, it is deleted from the potential keyframe image set. By continuously filtering and adjusting in this way, until the remaining keyframe images meet the minimum keyframe interval requirement, the final set of keyframe images used for subsequent processing is determined. This method improves the correlation between keyframes and the object being searched, solving the problems of rigid keyframe thresholds and irrelevance to the target object in traditional methods. Therefore, it effectively improves the quality and accuracy of keyframe images, laying a solid foundation for accurately locating the object.

[0068] This embodiment also relates to a method for finding items based on video playback. If a user loses the same item (such as a key) multiple times, historical item search data can be used to optimize the current search process. Specifically, such as... Figure 2 The server can store relevant information each time a user loses an item, including item characteristics, the search area, the search period, and the location where the item was eventually found. When a user loses the same item again, the server can first query this historical data to analyze common locations and time patterns where the user lost the item. Based on this analysis, the server can automatically adjust the search area and search period settings to better reflect the actual context in which the item was lost. Furthermore, the server can utilize item location information from historical data to help determine keyframe image selection criteria and image recognition parameters during the current search process, thereby improving the accuracy and efficiency of the search. In this way, the method proposed in this application can fully utilize historical data to provide users with a more personalized and intelligent item search service.

[0069] Specifically, if the characteristics of the item to be found appear more frequently than the first threshold (e.g., 3 times) within a preset time period (e.g., 3 days), the historical retrieval locations of the item are obtained. When the number of times the historical retrieval location is located in the current search area exceeds the second threshold (e.g., 2 times), the search video is determined by combining the historical retrieval location, the search time period, and the search area. For example, if the current search area is the home, and the previous items were found in the bedroom, the surveillance video in the bedroom can be directly used as the search video, without needing to process surveillance videos from other areas of the home. This greatly narrows the scope of video processing and improves the efficiency of finding items. Furthermore, if the historical retrieval locations show a clear temporal distribution pattern within the search time period, such as items always being lost and found in the bedroom at a certain time in the evening, the server can prioritize processing the surveillance video in the bedroom during that time period based on this pattern, further optimizing the search process.

[0070] In addition to leveraging historical item search data to optimize the identification of videos to be searched, the server can also adjust the selection criteria for keyframe images based on item location information from historical data. For example, if historical data shows that the item to be searched was usually stationary when it was lost, the server can lower the requirements for motion vectors in the keyframe images and instead focus more on areas in the image that highly match the item's features. Conversely, if historical data shows that the item to be searched was frequently moving when it was lost, the server can increase the requirements for motion vectors to ensure that the selected keyframe images accurately reflect the item's movement trajectory.

[0071] Furthermore, the server can utilize image recognition parameters from historical data to optimize the current image recognition process. Specifically, the server can analyze information such as the image recognition algorithm, parameter settings, and recognition accuracy used each time the item was successfully identified in historical data, extracting the optimal recognition strategy and parameter combination. When a user loses the same item again, the server can directly apply these optimal strategies and parameters for image recognition, thereby improving the accuracy and efficiency of the recognition.

[0072] By fully utilizing historical item-finding data, the method proposed in this application not only provides users with more personalized and intelligent item-finding services, but also continuously optimizes and improves its own item-finding performance. As user usage increases and historical data accumulates, the server's item-finding accuracy and efficiency will be further enhanced, bringing users a more convenient and efficient item-finding experience.

[0073] This embodiment also relates to a video backtracking-based item finding method. In S105, there may be multiple monitoring devices within the search area, corresponding to multiple monitoring videos. In this case, the item to be found may appear in multiple videos, meaning the item may not appear in a single video. Based on the timestamp and bounding box coordinates, the disappearance time and area of ​​the item in the current video can be determined. Then, based on this disappearance time and area, video frames matching the disappearance time and area are searched in the remaining videos. Specifically, the server uses the disappearance time of the item in a particular video as a benchmark, setting a reasonable time range before and after it. Within this time range, it searches frame by frame in the remaining videos for areas similar to or close to the disappearance area. By analyzing the image features within these areas, it determines whether the item exists. If a target matching the item's features is found, the location and time of the item's appearance in the remaining videos are determined. Next, the server integrates the appearance information of the item in all videos to further analyze its movement trajectory and possible search areas. In this way, even if the item appears in multiple search videos, the server can accurately determine its location and movement, providing users with more comprehensive and accurate results. At the same time, this cross-video search and analysis method improves the robustness of the search, enabling it to handle various complex monitoring environments and item loss scenarios.

[0074] If image recognition fails to identify the item, multimodal recognition can be used. For example, audio data synchronized with the video can be acquired and extracted to obtain keywords related to the item. Speech recognition technology can then be used to process the audio data, converting it into text. Natural language processing (NLP) is then used to analyze this text, extracting keywords potentially related to the item, such as the item's name or specific descriptive terms. These keywords are then combined with targeted image recognition on keyframe images or the entire video, expanding the recognition range or adjusting parameters to attempt to find the item matching the keywords. Furthermore, data from other modalities, such as sensor data from the monitoring scene (e.g., temperature, humidity, if relevant to the item's characteristics), can be combined to further assist in identification. This multimodal recognition approach fully utilizes available information, improving the success rate and accuracy of item retrieval when image recognition fails, providing users with a more reliable item-finding service.

[0075] Furthermore, the extracted audio features can be compared and analyzed with the sound features that the item to be found might emit. For example, if the item to be found is a mobile phone with a specific ringtone, the audio features are matched with the audio features of that phone's ringtone; if the item to be found is a toy with a specific sound, it is compared with the audio features of the toy's typical sound. If the audio features match successfully, combined with previously acquired video-related information, such as timestamps, the possible range of where the item to be found can be further narrowed down, helping to determine the location of the item. In addition to audio data, other modal data can also be combined. For example, if there are infrared sensors in the monitoring environment, infrared sensor data can be acquired. The infrared sensor data is analyzed to see if there is information such as heat source movement trajectories that match the size and shape of the item to be found. By fusing and analyzing infrared sensor data with video data, audio data, and other multimodal data, the location of the item to be found can be comprehensively determined. Multimodal recognition fully utilizes various possible data information, searching for the item to be found from multiple dimensions, greatly improving the success rate and accuracy of finding items in complex environments, and providing users with more reliable results.

[0076] This embodiment also relates to a video-based item retrieval method. After determining the search area, the method can combine the surveillance footage corresponding to that area at the current time to verify the search area in real time, confirming whether the item is still within that area. If the item is not found in the surveillance footage, the server will infer that it may have moved to an adjacent area based on the item's previous trajectory and speed, and further retrieve surveillance videos from that adjacent area for further searching. Simultaneously, the server can send information about the search area and the characteristics of the item to relevant staff or security personnel, requesting their assistance in the search to improve efficiency and accuracy. Furthermore, the server can record the search process and results, forming a search log for subsequent analysis and improvement.

[0077] After obtaining the timestamps and bounding box coordinates, when determining the search area for a missing item, the server can construct the object's motion trajectory within the video based on consecutive timestamps and bounding box coordinates. Specifically, the server connects the bounding box coordinates of the missing item in each frame in chronological order, forming a continuous motion path. This trajectory visually demonstrates the object's direction of movement and speed changes within the video. By analyzing the trajectory, the server can further infer the object's potential movement trend and final location. For example, if the trajectory shows the object moving steadily in a certain direction, the server can predict that it may continue moving in that direction, thus identifying that area as a key search area. Furthermore, the server can combine features such as inflection points and speed changes in the trajectory to further refine and adjust the search area. This approach allows for more accurate determination of the search area, improving both the efficiency and accuracy of the search process.

[0078] After obtaining the movement trajectory, the movement trajectory can be extrapolated to predict the possible direction of movement and destination area of ​​the item after it leaves the monitoring field of view. The possible direction of movement and destination area are then fused with the electronic map of the area to be searched to delineate multiple candidate search areas with different probabilities on the electronic map and provide the candidate search areas to the user.

[0079] This application provides users with more accurate and comprehensive guidance for finding items by meticulously analyzing movement trajectories and combining them with electronic maps. Specifically, it includes:

[0080] First, after acquiring the movement trajectory of the item to be searched, the server uses advanced trajectory prediction algorithms, such as machine learning-based motion models, to simulate the possible movement path of the item after it leaves the monitoring field of view. These algorithms comprehensively consider factors such as the item's speed, direction, acceleration, and possible obstacles, thereby generating multiple possible movement directions and destination areas.

[0081] Next, the server integrates these predictions with an electronic map of the area to be searched. This electronic map contains detailed geographical information, such as building layouts, road directions, and obstacle locations, which is crucial for identifying candidate search areas. By comparing and analyzing the predicted direction of movement and destination area with the electronic map, the server can delineate multiple candidate search areas with varying probabilities. These areas are ranked according to the probability of the item appearing within them, with the area having the highest probability considered the most likely location.

[0082] Finally, the server presents these candidate search areas to the user in an intuitive way. Users can choose to search in these areas based on their situation and needs. The server can also provide additional auxiliary information, such as a detailed description of each candidate search area and route planning to that area, to help users find their items more efficiently.

[0083] In this way, the method proposed in this application not only improves the accuracy and efficiency of finding items, but also provides users with a more personalized and intelligent item-finding service. Users no longer need to blindly search the entire search area, but can instead perform targeted searches based on the candidate search areas provided by the server, thereby greatly saving time and effort.

[0084] This embodiment also relates to an item finding device based on video playback, such as... Figure 3 The system includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0085] In response to receiving a lost item request from a user, the system determines the item characteristics, the search area, and the search time period based on the lost item request; wherein the item characteristics include at least one of category characteristics, appearance characteristics, and image characteristics.

[0086] Based on the time period and area to be searched, the video to be searched is determined;

[0087] Based on the motion vectors of feature points within each video frame of the video to be searched, the keyframe images of the video to be searched are determined.

[0088] Based on the features of the item, image recognition is performed on the keyframe images to determine the bounding box coordinates and timestamp of the item to be found in each keyframe image;

[0089] Based on the timestamp and the bounding box coordinates, the search area for the item to be found is determined.

[0090] This embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0091] In response to receiving a lost item request from a user, the system determines the item characteristics, search area, and search time period of the item to be found based on the request. The item characteristics include at least one of category features, appearance features, and image features. Based on the search time period and search area, the system determines the video to be found. Based on the motion vectors of feature points within each video frame of the video to be found, the system determines the keyframe images of the video to be found. Based on the item characteristics, the system performs image recognition on the keyframe images to determine the bounding box coordinates and timestamp of the item to be found in each keyframe image. Based on the timestamp and bounding box coordinates, the system determines the search area of ​​the item to be found.

[0092] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0093] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0100] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for finding items based on video playback, characterized in that, include: Based on the received lost item request, determine the characteristics of the lost item, the area to be searched, and the time period to be searched; The item features mentioned above include at least one of category features, appearance features, and image features; Based on the time period to be searched and the area to be searched, the video to be searched is determined; Based on the motion vectors of feature points in each frame of the video to be searched, the keyframe images of the video to be searched are determined. Specifically, this includes: traversing the input video to be searched; for each frame image to be detected in the video to be searched, determining the motion vector features of all feature points in the frame image to be detected and the next frame image, wherein the motion vector features include the average motion amplitude and the proportion of feature points whose motion amplitude is greater than a preset threshold; determining the historical motion vector features of the frame image to be detected within a preset historical time period, and using the historical motion vector features as a measurement threshold for the frame image to be detected; when the motion vector features of the frame image to be detected are higher than the measurement threshold, the frame image to be detected is used as the keyframe image of the video to be searched. Based on the features of the item, image recognition is performed on the keyframe images to determine the bounding box coordinates and timestamp of the item to be found in each keyframe image; Based on the timestamp and the bounding box coordinates, the search area for the item to be found is determined.

2. The method for finding items based on video backtracking according to claim 1, characterized in that, The method further includes: determining feature points in each frame of the video to be searched before determining the keyframe images, specifically: Iterate through each frame of the input video to be searched. For each pixel to be detected in each frame, determine the first gray intensity of the pixel to be detected and the second gray intensity set of other pixels within a preset range of the pixel to be detected. If all second grayscale intensities within the second grayscale intensity set are higher than the sum of the first grayscale intensity and a preset threshold, or are all lower than the difference between the first grayscale intensity and the preset threshold, then the pixel to be detected is taken as the feature point; and / or, Create a two-dimensional uniform grid on the video frame image; The pixel positions of all intersections or center points of the grid are used as the feature points.

3. The method for finding items based on video backtracking according to claim 1, characterized in that, The process of determining the keyframe images of the video to be searched also includes: receiving the minimum keyframe interval set by the user, and determining the frame interval between each adjacent keyframe image.

4. The method for finding items based on video backtracking according to claim 3, characterized in that, After determining the frame interval between each adjacent keyframe image, the method further includes: in response to the frame interval between adjacent keyframe images being less than the minimum keyframe interval, performing image recognition on two adjacent keyframe images based on the item features to determine the image confidence of the two keyframe images; Delete keyframe images with an image confidence level lower than the confidence threshold until the frame interval between any two adjacent keyframe images is greater than the minimum keyframe interval.

5. The method for finding items based on video backtracking according to claim 1, characterized in that, Based on the time period and the region to be searched, the video to be searched is determined, specifically including: In response to the fact that the number of times the item characteristics of the item to be found appear within a preset time period is higher than the first time threshold, the historical retrieval location of the item to be found is obtained; When the number of times the historical retrieval location is located in the area to be searched is higher than the second threshold, the video to be searched is determined based on the historical retrieval location, the time period to be searched, and the area to be searched. When the number of times the historical retrieval location is located in the area to be searched is not higher than the second number threshold, the video to be searched is determined based on the time period to be searched and the area to be searched.

6. The method for finding items based on video backtracking according to claim 1, characterized in that, Determining the search area for the item to be found specifically includes: Based on continuous timestamps and bounding box coordinates, the motion trajectory of the object to be found in the video to be found is constructed; Extrapolate the motion trajectory to predict the possible direction of movement and destination area of ​​the object after it leaves the monitoring field of view; The possible directions of movement and destination areas are merged with the electronic map of the area to be searched to delineate multiple candidate search areas with unequal probabilities on the electronic map, and the candidate search areas are provided to the user.

7. The method for finding items based on video backtracking according to claim 1, characterized in that, The method also supports determining the search area for the item across monitoring devices, specifically including: Based on the timestamp and the bounding box coordinates, the disappearance time and disappearance area of ​​the item to be found in the video to be found are determined; In response to the disappearance area being within the monitoring range of other monitoring devices, based on the disappearance time and the disappearance area, subsequent videos to be searched by other monitoring devices are determined; Image recognition is performed based on key frame images of the subsequent video to be searched until the final disappearance area is no longer within the monitoring range, or the search area of ​​the item to be searched is determined. Send the final disappearance area or the search area to the user.

8. The method for finding items based on video backtracking according to claim 1, characterized in that, The process of performing image recognition on the keyframe image based on the features of the item includes: In response to the failure to identify the item by image recognition, audio data synchronized with the video being searched is acquired; Based on natural language processing technology, keywords related to the item to be found are identified from the audio data; Locate the time point when the keyword appears, and extract the video frame at the corresponding time point as a supplementary keyframe; Based on the characteristics of the item, image recognition is performed on the supplementary keyframes to locate the item to be found.

9. The method for finding items based on video backtracking according to claim 1, characterized in that, If a user loses the same item multiple times, historical lost item data will be used to optimize the current search process, specifically including: The system automatically adjusts the search area and search time period based on historical search data, and uses the item location information in the historical search data to adjust the key frame image screening criteria and image recognition parameters in the current search process.

10. A video-based item retrieval device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method as claimed in any one of claims 1-9.

Images