Video Processing and Information Query Method, Device, System and Storage Medium

Abstract

Embodiments of the present application provide a video processing and information query method, device, system, and storage medium. In the embodiments of the present application, various objects appearing in video data are structured according to object categories to obtain structured data of various objects. By combining the feature vectors of the objects included in the structured data and filtering the structured data, the structured data of mis-identified objects can be filtered out, so as to improve the accuracy of the structured data, provide an accurate data basis for subsequent operations such as searching based on structured data, image search by image, and trajectory tracking, and improve the accuracy and effectiveness of subsequent operations.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing and information query method, device, system and storage medium. Background Art

[0002] In video scenarios, the amount of video storage is constantly increasing, which brings huge challenges to video search and video monitoring, so video structuring technology has emerged. Video structuring is a technology that extracts targets from video data in a structured manner based on a neural network model. Compared with the original video, structured data can not only save storage space, but also greatly improve the efficiency of search and troubleshooting.

[0003] However, due to the generalization limitations of the neural network model itself, that is, the model cannot fit the video data it has never seen well, resulting in a large amount of misidentified structured data in daily applications. For example, moving trash cans, plants as tall as people, or roadside sentry booths are easily misidentified as pedestrians by the model.

[0004] In the existing technology, a large amount of video data from new scenes is usually collected and used as training samples to retrain the model, trying to solve the problem from the aspects of enriching training samples and model capabilities. However, retraining the model may introduce uncertainty and still have limitations for video data with unknown attributes, and cannot fundamentally solve the problem. Summary of the invention

[0005] Multiple aspects of the present application provide a video processing and information query method, device, system and storage medium to improve the accuracy of video structured processing and improve the efficiency of search and troubleshooting based on structured data.

[0006] The embodiment of the present application provides a video processing method, including: receiving video data uploaded by a client, wherein the video data includes objects of at least one category; performing structured processing on the video data to obtain structured data under at least one category, wherein each structured data corresponds to an object under the category to which it belongs and includes a feature vector of the object; for a target category, based on the feature vector of the object included in the structured data under the target category, filtering out the structured data corresponding to the misidentified object under the target category; wherein the target category is any category of the at least one category, and the misidentified object refers to an object in the video data that is misidentified as the target category.

[0007] An embodiment of the present application further provides a data filtering method, including: receiving structured data of at least one category output by a video processing device, each piece of structured data corresponding to an object under its corresponding category and including a feature vector of the object; for a target category, filtering out the structured data corresponding to mis-identified objects under the target category based on the feature vectors of the objects included in the structured data under the target category; outputting the unfiltered structured data under the target category to a storage system corresponding to the video processing device for storage; wherein, the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

[0008] An embodiment of the present application further provides a video processing method, including: receiving a surveillance video in a specified environmental space uploaded by a video capture device, the video capture device being installed in the specified environmental space; performing structured processing on the surveillance video to obtain structured data of at least one object category, each piece of structured data corresponding to an object under its corresponding object category and including a feature vector of the object; for a target category, filtering out the structured data corresponding to mis-identified objects under the target category based on the feature vectors of the objects included in the structured data under the target category; wherein, the target category is any one of the at least one object category, and the mis-identified object refers to an object in the surveillance video that is mis-identified as the target category.

[0009] An embodiment of the present application further provides a video processing method, including: uploading video data to a video processing device to request structured processing of the video data, the video data including objects of at least one category; receiving unfiltered structured data of at least one category returned by the video processing device and structured data filtered according to the feature vectors of the objects included in the structured data; correspondingly displaying the unfiltered and filtered structured data of the at least one category for the user to know the mis-identified objects of the at least one category.

[0010] An embodiment of the present application further provides an information query method, including: sending a first query request to request querying information of a first object, the first object belonging to a target category; receiving first information returned according to the first query request, the first information being queried from the unfiltered structured data under the target category; in the case of determining that the first information does not belong to the first object, sending a second query request to request re-querying information of the first object; receiving second information returned according to the second query request and displaying the second information as the information of the first object, the second information being queried from the structured data filtered according to the feature vectors of the objects included in the structured data under the target category.

[0011] An embodiment of the present application further provides a data processing device, including: a processor and a memory storing a computer program; the processor is configured to execute the computer program for: receiving video data uploaded by a client, where the video data includes at least one category of objects; performing structured processing on the video data to obtain structured data under at least one category, each structured data corresponding to an object under its respective category and including a feature vector of the object; for a target category, based on the feature vectors of the objects included in the structured data under the target category, filtering out the structured data corresponding to mis-identified objects under the target category; where the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

[0012] An embodiment of the present application further provides a data processing system, including: a client, a video processing device, and a storage system; where the client is configured to provide video data, and the video processing device is configured to perform structured processing on the video data provided by the client, generate structured data, and store the structured data in the storage system; when performing structured processing on video data, the video processing device is configured to:

[0013] receive the video data uploaded by the client, where the video data includes at least one category of objects; perform structured processing on the video data to obtain structured data under at least one category, each structured data corresponding to an object under its respective category and including a feature vector of the object; for a target category, based on the feature vectors of the objects included in the structured data under the target category, filtering out the structured data corresponding to mis-identified objects under the target category; where the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

[0014] In the embodiment of the present application, by performing structured processing on various objects appearing in the video data according to the object category, obtaining structured data of various objects, and combining the feature vectors of the objects included in the structured data to filter the structured data, the structured data corresponding to mis-identified objects can be filtered out, so as to improve the accuracy of the structured data, provide an accurate data basis for subsequent operations such as searching based on structured data, image search by image, and trajectory tracking, and improve the accuracy and effectiveness of subsequent operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0016] Figure 1a It is a schematic structural diagram of a video processing system provided by an embodiment of the present application;

[0017] Figure 1b It is a schematic structural diagram of another video processing system provided by an embodiment of the present application;

[0018] Figure 2 It is a schematic structural diagram of yet another video processing system provided by an embodiment of the present application;

[0019] Figure 3a It is a flowchart of a video processing method provided by an embodiment of the present application;

[0020] Figure 3b It is a flowchart of a data filtering method provided by an embodiment of the present application;

[0021] Figure 3c It is a flowchart of another video processing method provided by an embodiment of the present application;

[0022] Figure 3d It is a flowchart of an information query method provided by an embodiment of the present application;

[0023] Figure 4 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application. Detailed implementation manners

[0024] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] An embodiment of the present application provides a video processing system, as Figure 1a shown. The video processing system includes: a client 10, a video processing device 20, and a storage system 30. The client 10 and the video processing device 20, as well as the video processing device 20 and the storage system 30, can be communicatively connected in a wired or wireless manner.

[0026] In this embodiment, the client 10 is responsible for providing video data, which can be a storage device for storing video data, a video server for processing video data, or a video capture terminal capable of capturing video data, such as a camera, an electronic eye, etc. Physically, the video processing device 20 can be a conventional server, a cloud server, a virtual machine, or a server array, etc. The video processing device 20 can provide video structuring services for the client 10 and is responsible for structuring the video data provided by the client 10. As Figure 1a shown in ①, the client 10 can upload video data to the video processing device 20, and the video processing device 20 can receive the video data uploaded by the client 10.

[0027] In this embodiment, the objects that can appear in the video data are classified, and the classification criteria and classification granularity are not limited. For example, the objects can be classified into pedestrians, vehicles, animals, plants, etc. Further, pedestrians can be classified into children, the elderly, and adults, or further classified into male pedestrians and female pedestrians, etc. For vehicles, further classification can be made according to information such as the type, brand, or manufacturer of the vehicle. That is to say, the video data provided by the client 10 includes at least one category of objects. Based on this, as Figure 1a shown in ②, after receiving the video data uploaded by the client 10, the video processing device 20 can perform structuring processing on the video data to obtain structured data under at least one category. In this embodiment, the video processing device 20 can use an AI-based video algorithm to perform structuring processing on the video data. The AI-based video algorithm can be a neural network-based video algorithm. Among them, the neural network used can be but not limited to: RNN, CNN, STLM, etc. In this embodiment, the AI-based video algorithm can perform structuring processing on at least one category of objects. Which object categories the video algorithm specifically supports can be set according to application requirements during the training process. For example, a video algorithm that can perform structuring processing on pedestrians can be pre-trained, a video algorithm that can perform structuring processing on vehicles can be pre-trained, or a video algorithm that can simultaneously perform structuring processing on pedestrians and vehicles can be trained, etc.

[0028] Among them, the structured processing of video data mainly refers to the process of forming structured data of each object under at least one category contained in the video data, and the process includes: video decoding, target detection and target tracking, and structuring and other processing. In detail, after receiving the video data uploaded by the client 10, if the video data is encoded, the video processing device 20 first decodes the video data, and then performs target detection on the decoded video data to obtain objects under at least one category; then, for each object under each category, track the video frame containing the object in the video data and track the position of the object in the video frame and other information; further, the video frame is structured in combination with the position information of the object in the video frame to obtain the structured data of the object. Among them, each structured data obtained by the video processing device 20 has a category attribute, corresponds to an object under the category to which it belongs, and includes a feature vector of the object, and the feature vector of the object is the result of vectorizing the features of the object. In the embodiment of the present application, in addition to the feature vector of the object, the structured data of each object may also include the attribute information of the object extracted from the video frame where the object is located. Among them, the attribute information of the object will also be different depending on the type of the object. Taking a person as an example, the attribute information of an object includes the category information of the object (such as a pedestrian), the time and location of the object appearing in the video, and the object's own attributes, such as whether it wears a hat, clothing color, height, age, etc. Taking a vehicle as an example, the attribute information of an object includes the category information of the object (such as a vehicle), the time and location of the object appearing in the video, and the object's own attributes, such as the vehicle's color, size, model, license plate number, etc.

[0029] In the process of structuring video data, objects in one category may be mistakenly identified as objects in another category due to the generalization limitations and accuracy of the algorithm. In this way, the final structured data will contain erroneous structured data. For example, in the process of structuring pedestrians in traffic monitoring videos, other objects with the same height and similar appearance as pedestrians, such as trash cans, sentry boxes, trees or other moving objects, may be mistakenly identified as pedestrians. In this case, there will be some structured data that are not actually pedestrians in the structured data of pedestrians obtained by structured processing. However, with the development of video structuring technology, many applications in the city brain, such as security, tourism, health, and transportation, require real-time structured processing of pedestrians, vehicles and other objects in massive videos, and search, image search, trajectory tracking and other operations based on structured data. It can be seen that the accuracy of structured data directly affects the accuracy and effectiveness of subsequent operations that rely on structured data, such as search, image search, and trajectory tracking.

[0030] In view of the above problems, in this embodiment, after obtaining the structured data of at least one category, the structured data of any category in the at least one category can also be filtered according to application requirements, such as Figure 1a shown in ② of Figure 1a . For the convenience of description and distinction, the object category that needs to be filtered is called the target category, and the target category can be any category in the at least one category. For the target category, the video processing device 20 can filter out the structured data of the mis-identified objects corresponding to the target category based on the feature vectors of the objects included in the structured data of each object under the target category, so as to obtain the accurate structured data under the target category; among them, the mis-identified object is an object that identifies an object of other categories as the target category. For example, taking the pedestrian category as an example, trash cans, sentry boxes, or trees that are mis-identified as pedestrians are mis-identified objects, and the structured data of these mis-identified objects can be filtered out from the structured data of the pedestrian category, so as to obtain the structured data of pedestrians with higher accuracy, which is beneficial to the accuracy and effectiveness of subsequent operations such as searching based on the structured data of pedestrians, image search by image, or trajectory tracking.

[0031] Optionally, as Figure 1a shown in ③ of Figure 1a , after filtering out the structured data of the corresponding mis-identified objects, the video processing device 20 can output the unfiltered structured data under the target category to the storage system 30 for storage, providing a data basis for subsequent operations such as searching based on structured data, image search by image, and trajectory tracking. For example, as Figure 1a shown in ④ of Figure 1a , the client 10 can send a query request to the storage system 30, and the query request can include the identification information of the target object. Taking a vehicle as an example, the identification information of the target object can be the license plate number; taking a person as an example, the identification information of the target object can be the head portrait or photo of the object, etc. Of course, the identification information of the target object can also be the structured data of the target object obtained by performing structured processing on the video data containing the target object. Further, as Figure 1a shown in ⑤ and ⑥ of Figure 1a , the storage system 30 can query the structured data corresponding to the identification of the target object from the stored structured data according to the identification information of the target object, and return the query result to the client 10 when the query is successful. In terms of the implementation form, there is no limitation on the content returned by the storage system 30 to the client 10. For example, it can return the structured data converted into the photo of the target object, or directly return the structured data corresponding to the target object. It should be noted that the client 10 can directly send a query request to the storage system 30, or send a query request to the video processing device 20, and the video processing device 20 can query from the storage system 30. Of course, in addition to the client 10 can send a query request to the storage system 30 or the video processing device 20, other devices with query permissions can also send query requests to the storage system 30 or the video processing device 20 according to query requirements. InFigure 1a is not shown.

[0032] Further optionally, the video processing device 20 may also output the unfiltered structured data and the structured data before filtering in the target category to the storage system for storage. In this way, when it is necessary to query the structured data before filtering in the target category, it can be queried from the structured data before filtering in the target category stored in the storage system 30; or, when it is necessary to compare the structured data in the target category before and after filtering, the unfiltered structured data and the structured data before filtering in the target category can be obtained from the storage system 30 and compared; or, when it is necessary to view the structured data of the mis-identified object corresponding to the target category, the unfiltered structured data and the structured data before filtering in the target category can be obtained from the storage system 30, and the structured data filtered in the target category, that is, the structured data of the mis-identified object corresponding to the target category, can be obtained by comparison.

[0033] Further optionally, after the video processing device 20 identifies the structured data of the mis-identified object corresponding to the target category, it may also mark the structured data of the mis-identified object corresponding to the target category, and output the structured data with and without marks in the target category correspondingly. For example, the structured data with and without marks in the target category may be displayed correspondingly. Among them, the structured data with marks is the structured data of the mis-identified object that has been marked; on the contrary, the structured data without marks refers to the correctly identified structured data in the target category. By marking the structured data of the mis-identified object corresponding to the target category, it is convenient and intuitive to understand the mis-identified structured data in the target category.

[0034] Further optionally, after the video processing device 20 obtains the unfiltered and filtered structured data in at least one category, it may also return the unfiltered and filtered structured data in at least one category to the client 10; the client 10 receives the unfiltered and filtered structured data in at least one category returned by the video processing device 20, as shown in ⑦ of Figure 1a ; and display the unfiltered and filtered structured data in at least one category correspondingly, so that the user can know the mis-identified objects in at least one category, as shown in ⑧ of Figure 1a .

[0035] In this embodiment, the client 10 refers to the end that can provide video data and requires the video processing device 20 to provide video structuring services, and the implementation form of the client 10 is not limited. In an alternative embodiment, the client can be implemented as a video capture device 40 installed in a specified environmental space, responsible for capturing surveillance videos in the specified environmental space, and the surveillance videos may include at least one type of object. Among them, according to the different specified environmental spaces, the objects and object categories in the surveillance videos will also be different. For example, taking a shopping mall as an example, the objects in the surveillance video may include consumers shopping, shelves, cash registers, etc.; taking a traffic space as an example, the objects in the surveillance video may include vehicles, pedestrians, road signs, trash cans by the roadside, etc. Figure 1b Taking the traffic space as the specified environmental space as an example, as Figure 1b shown in ①, the video capture device 40 is a surveillance camera installed at a traffic intersection, which can capture surveillance videos of passing vehicles and pedestrians. After the video capture device 40 captures the surveillance video, it can upload it to the video processing device 20.

[0036] Furthermore, as Figure 1b shown in ②, after the video processing device 20 receives the surveillance video in the traffic space uploaded by the video capture device 40, it can perform structuring processing on the surveillance video to obtain structured data under at least one object category. Among them, each structured data corresponds to an object under its respective object category and includes the feature vector of the object. Considering that during the structuring process, an object of one category may be misidentified as an object of another category, which means that there may be structured data misidentified as an object of this category in the structured data under a certain category. Therefore, after obtaining the structured data under at least one category, it is also possible to perform filtering processing on the structured data of any category in at least one category as needed, as Figure 1b shown in ②. Specifically, for the target category, the video processing device 20 can filter out the structured data corresponding to the misidentified objects under the target category based on the feature vectors of the objects included in the structured data under the target category; where the target category is any object category in at least one object category, and the misidentified object refers to an object of another category in the surveillance video that is misidentified as an object of the target category. Further, as Figure 1b shown in ③, the video processing device 20 can output the unfiltered structured data under the target category to the storage system 30 for storage, providing a data basis for subsequent operations such as searching based on structured data, image search by image, and trajectory tracking. Or, it is also possible to output the unfiltered structured data under the target category and the structured data before filtering under the target category to the storage system for storage; or, it is also possible to mark the structured data corresponding to the misidentified objects under the target category and output the structured data with and without marks under the target category correspondingly.

[0037] In this embodiment, the specified environmental space is not limited. For example, it may be, but is not limited to: the interior space of a public transportation vehicle, or an office park, or an urban road traffic space, or a self-service supermarket space, and a campus space, etc. The working principle of the embodiments of the present application will be described below with specific scenario examples.

[0038] Scenario Example 1: In a public transportation vehicle, for the personal and property safety of passengers, surveillance cameras can be installed at the door and in convenient viewing positions in the carriage. During the process of passengers getting on and off the vehicle, the surveillance camera installed at the door can collect video information of passengers getting on and off the vehicle. The driver can view the status of passengers getting on and off the vehicle according to the surveillance video to avoid safety accidents caused by opening or closing the door in advance. On the other hand, the surveillance camera in the carriage can also record the video information of passengers during the ride in real time and upload the recorded video data to the video processing device. After the video processing device performs structured processing and filtering on the surveillance video uploaded by the surveillance camera, the target structured data can be stored in the storage system. In the case of theft or an accident in a public transportation vehicle, a query request can be sent to the video processing device. The query request can carry information about the public transportation vehicle and date information. The video processing device, according to the information of the public transportation vehicle and date information, retrieves the structured data of each object in the video data captured on the public transportation vehicle on the date indicated by the date information from the storage system, and searches for theft suspects or analyzes the cause of the accident, etc. based on the surveillance video corresponding to this structured data.

[0039] Scenario Example 2: In an office park, to protect the park environment, the safety of office workers and property in the park, surveillance cameras can be installed around the green plant areas in the office park and at the entrances and exits of the park and the office building. The surveillance cameras can collect video of pedestrians and vehicles in the park and send the collected surveillance video to the video processing device. After the video processing device performs structured processing and filtering on the surveillance video uploaded by the surveillance camera, the target structured data can be stored in the storage system. When security guards or other supervisors find that someone is damaging the park environment, or theft, robbery, or other incidents occur, a query request can be sent to the video processing device. The query request can carry the location information of the park and date information. The video processing device, according to the location information of the park and date information, retrieves the structured data of each object in the video data captured in the park on the date indicated by the date information from the storage system, and searches for those who damage the environment, steal, or rob based on the surveillance video corresponding to this structured data. Since there are multiple surveillance cameras in the park, when sending a query request to the data storage system, the identification information of the surveillance camera can also be specified to quickly obtain the target video and search for those who damage the environment or steal or rob.

[0040] Scenario Example 3: In the urban road traffic space, in order to ensure the orderly movement of vehicles and avoid traffic accidents, surveillance cameras or electronic eyes can be installed on both sides of the road or at traffic posts. The cameras or electronic eyes can capture the driving videos of passing vehicles in real time and send the captured videos to the video processing device. After structuring and filtering the surveillance videos uploaded by the surveillance cameras or electronic eyes, the video processing device can store the target structured data in the storage system. When a traffic accident occurs at a certain intersection and there is a need to hold someone accountable or find a hit-and-run vehicle, traffic supervisors can send a query request to the storage system through the management device. The query request can carry the location information of the accident and the time information of the accident occurrence. The video processing device retrieves the structured data of each object in the video data captured at the accident location and at the accident time from the storage system according to the location information and time information of the accident occurrence. Based on the surveillance videos corresponding to these structured data, the hit-and-run vehicle can be found to analyze the cause of the accident. If the hit-and-run vehicle escapes, the escape route of the hit-and-run vehicle can also be analyzed from the structured data corresponding to the video data of the vehicle captured after the accident time, which can be used to pursue the hit-and-run driver.

[0041] The embodiment of the present application also provides another video processing system, as Figure 2 shown, the video processing system includes: a client 10, a video processing device 21, a video filtering device 22, and a storage system 30. Between the client 10 and the video processing device 21, and between the video processing device 21 and the video filtering device 22 and the storage system 30, they can be communicatively connected in a wired or wireless manner.

[0042] In this video processing system, the client 10 is responsible for providing video data, which can be a storage device for storing video data or a video acquisition terminal capable of collecting video data, such as cameras, electronic eyes, etc. The video processing device 21 can provide video structuring services for the client 10 and is responsible for structuring the video data provided by the client 10. Physically, the video processing device 21 can be a conventional server, a cloud server, a virtual machine, or a server array, etc. As Figure 2 shown in ①, the client 10 can upload the video data to the video processing device 21, and the video data includes at least one type of object.

[0043] Further, as Figure 2As shown in ②, the video processing device 21 is responsible for performing structured processing on the video data to obtain structured data of at least one category. The detailed process of structured processing can be found in the above-mentioned embodiment, which will not be repeated here. Since in the process of structured data processing, objects under one category may be mistakenly identified as objects under another category due to problems such as the generalization limitations and accuracy of the algorithm, erroneous structured data will exist in the structured data finally obtained, affecting the accuracy of the structured data. Therefore, when generating structured data, the structured data can be filtered, and the structured data corresponding to the misidentified object category can be filtered out to improve the accuracy of the recognition result. Therefore, in the system of the present embodiment, a video filtering device 22 is added to filter the structured data. In terms of physical implementation, the video filtering device 22 can be a conventional server, a cloud server, a virtual machine or a server array, without limitation.

[0044] Further, if Figure 2 As shown in ③, after obtaining at least one category of structured data, the video processing device 21 can send at least one category of structured data to the video filtering device 22, and the video filtering device 22 receives at least one category of structured data output by the video processing device 21. Each structured data corresponds to an object in the category to which it belongs, and includes a feature vector of the object. Figure 2 As shown in ④, for a target category, the video filtering device 22 can filter out the structured data corresponding to the misidentified object under the target category based on the feature vector of the object included in the structured data under the target category, wherein the target category is any category of at least one category, and the misidentified object refers to an object in the video data that is mistakenly identified as a target category.

[0045] Further, if Figure 2 As shown in ⑤, the video filtering device 22 can output the structured data that is not filtered out under the target category to the storage system 30 corresponding to the video processing device 21 for storage, providing a data basis for subsequent operations such as search based on structured data, image search, and trajectory tracking. Figure 2 As shown in ⑥, ⑦, and ⑧, when it is necessary to obtain structured data, the client 10 may send a query request to the storage system 30, which may include the target category object identifier or attribute information. The storage system 30 may query the stored structured data based on the information and return the query result to the client 10. Since the storage system 30 stores structured data after the video filtering device 22 filters out misidentified objects, the result obtained by the client 10 is more accurate.

[0046] Further optionally, the video filtering device 22 may also output the unfiltered structured data and the structured data before filtering under the target category to the storage system for storage; alternatively, it may also mark the structured data corresponding to the mis-identified objects under the target category, and output the structured data with and without marks under the target category accordingly.

[0047] Further optionally, after obtaining the unfiltered and filtered structured data under at least one category, the video filtering device 22 may also return the unfiltered and filtered structured data under at least one category to the client 10; the client 10 receives the unfiltered and filtered structured data under at least one category returned by the video filtering device 22, as Figure 2 shown in; and display the unfiltered and filtered structured data under at least one category accordingly, for the user to know the objects mis-identified under at least one category, as Figure 2 shown in.

[0048] As Figure 2 shown, in the embodiment of the present application, the video processing system may further include a query end 40, and the query end 40 may be a video acquisition device capable of acquiring videos, or a terminal device performing other processing operations based on video information. In some embodiments, due to the shooting angle problem, the acquired video may have a certain gap from the target video. If the query end 40 is a terminal device for processing video information, the results obtained by processing based on these imperfect video information may be inaccurate. Therefore, when the query end 40 needs to obtain detailed video information, as Figure 2 shown in ⑨ in, the query end 40 may send the video data to be queried to the video processing device 21. Further, as Figure 2 shown in ② in, the video processing device 21 may perform structured processing on the video data sent by the query end 40 to obtain the structured data of the object to be queried. In order to obtain more detailed video information, as Figure 2 shown in ⑩ in, the video processing device 21 may perform a query operation on the storage system 30 according to the structured data of the object to be queried, obtain the structured data with the highest similarity to the structured data to be queried, obtain richer information of the object to be queried from this structured data, improve the structured data of the object to be queried, and as Figure 2 shown in ⑪ in, return the improved structured data to the query end 40 for the query end 40 to perform corresponding processing. For example, a certain video data is a side image of a vehicle, and the license plate number of the vehicle is not captured in the image. The query end 40 may send the video data to the video processing device 21. Through the video processing device 21 performing structured processing and query on the video data, the structured data corresponding to the vehicle in the video data that has been stored can be obtained, and the license plate number information of the vehicle can be obtained therefrom.

[0049] In the above embodiments, when a query request is initiated by the query end 40, the client 10, or the video processing device 21, a query is directly performed on the filtered structured data, and a query result with a relatively high accuracy is returned. In addition, when the query end 40, the client 10, or the video processing device 21 initiates a query request for the first time, a query can be performed on the unfiltered structured data, and a first query result is returned; if the first query result does not meet the requirements, the query end 40, the client 10, or the video processing device 21 can also initiate a second query request. At this time, a secondary query can be performed on the filtered structured data, and a second query result is returned to improve the accuracy of the query result. Among them, the filtered structured data can be obtained in advance, or after receiving the second query request, in real time according to the second query request, based on the feature vectors of the objects included in the unfiltered structured data, the unfiltered structured data is filtered to obtain the filtered structured data.

[0050] In the above embodiments of the present application, the implementation of filtering out the structured data of the mis-identified objects corresponding to the target category by the video processing device 21 or the video filtering device 22 based on the feature vectors of the objects included in the structured data under the target category is not limited. The following is an example:

[0051] In an alternative embodiment, the video processing device 21 or the video filtering device 22 can obtain the center vector of the objects under the target category according to the feature vectors of the objects included in the structured data under the target category; according to the center vector and the feature vectors of the objects included in the structured data, the mis-identified objects under the target category are identified, and the structured data of the mis-identified objects corresponding to the target category is filtered out. Among them, the center vector is a feature vector that measures the similarity degree of each object to the objects of the target category, and can be denoted as X1.

[0052] In the embodiments of the present application, the method for determining the center vector X1 is not limited. In an alternative embodiment, the mean value of the feature vectors of the objects included in the structured data under the target category can be calculated, and the obtained mean vector is used as the center vector X1. For example, the structured data under the target category is used as the first sample set S1. Assume that the number of structured data in the first sample set S1 is n, that is, n objects are identified, and the feature vector of each object is denoted as F i , then the center vector X1 can be expressed as: , where the feature vector F i can be a 256-dimensional or 512-dimensional floating-point number array, or other dimensions and other data types can be selected according to requirements, which are not limited herein. After obtaining the center vector X1, the feature vector F of each object in the first sample set S1 can be calculatedi The distance from the central vector X1, which is used to measure the similarity between each object and the objects of the target category. Based on these distances, mis-identified objects under the target category are identified, and then the structured data corresponding to the mis-identified objects is filtered out.

[0053] In another alternative embodiment, to make the filtered result more accurate, the mean vector can be obtained by calculating the mean of the feature vectors of the objects included in each structured data under the target category; based on the distances between the feature vectors of the objects included in each structured data under the target category and this mean vector, the feature vectors of the top N objects closest to this mean vector are selected; the mean of the feature vectors of the top N objects is calculated as the central vector of the objects under the target category; where N is an integer greater than or equal to 2. For example, the structured data under the target category is used as the first sample set S1. Assume the number of structured data in the first sample set S1 is n, that is, n objects are identified, and the feature vector of each object is denoted as F i , then the mean vector X0 can be expressed as: ; Based on the mean vector X0, obtain the feature vector F i Some objects close to the mean vector X0 are used as the second sample set S2. For example, select the feature vectors F of the N objects with the closest distance to the mean vector X0 i to form the second sample set S2; then, calculate the mean of the feature vectors of the objects in the second sample set S2 to obtain the central vector X1. In the case of obtaining the central vector X1, the distance between the feature vector of each object in the first sample set S1 and the central vector X1 can be calculated, and based on these distances, mis-identified objects under the target category are identified, and then the structured data corresponding to the mis-identified objects is filtered out.

[0054] In this embodiment, the implementation method of the distance between the feature vector of the object included in each structured data and the mean vector X0 is not limited. For example, it can be the Euclidean distance or the cosine distance, etc. Further, the value standard of N is not limited either. For example, it can be appropriately valued according to the number of objects in the first sample set S1. Assume that according to the distances between the feature vectors of each object and the mean vector X0 sorted from largest to smallest, the feature vectors of the top 70% or 80% are taken from the sorting result to form the second sample set S2.

[0055] When filtering out the structured data corresponding to the mis-identified objects, a first distance threshold between the feature vector of the object and the central vector X1 can be set. By calculating the distances between the feature vectors of the objects included in each structured data and the central vector X1, and taking the objects with a distance greater than the first distance threshold from the central vector X1 as the mis-identified objects under the target category; then the structured data corresponding to the mis-identified objects is filtered out.

[0056] Further optionally, an object set Sn can also be maintained for the target category, and the object set Sn is used to store the feature vectors of the mis-identified objects under the target category. Based on this, after identifying a mis-identified object whose distance from the central vector X1 is greater than the first distance threshold, the mis-identified object can also be added to the object set Sn as a sample set for further secondary filtering of the objects whose distance from the central vector X1 is less than or equal to the first distance threshold. Optionally, the mis-identified object whose distance from the central vector X1 is greater than the first distance threshold can be directly added to the object set Sn. Or, when adding the mis-identified object whose distance from the central vector X1 is greater than the first distance threshold to the object set Sn, the similarity between the mis-identified object and each object in the object set Sn can be calculated according to the feature vector of the mis-identified object. If the similarity between the mis-identified object and each object in the object set Sn is less than the similarity threshold, the mis-identified object is added to the object set Sn; otherwise, the mis-identified object is discarded. In this embodiment, the size of the similarity threshold is not limited and can be appropriately adjusted according to the recognition accuracy. For example, if the recognition accuracy is low and only objects with significant features need to be identified, the similarity threshold can take a relatively large value, such as 0.7 or 0.8, to reduce the redundant data in the object set Sn; if the recognition accuracy is high and objects with subtle features that are difficult to identify need to be identified, the similarity threshold can take a relatively small value, such as 0.4 or 0.5, to enrich the number of negative samples in the object set Sn and provide a basis for further filtering mis-identified objects.

[0057] Further optionally, in order to reduce the probability that there are still mis-identified objects after the first filtering, the structured data after the first filtering can be filtered again. That is, for an object whose feature vector is less than or equal to the first distance threshold from the central vector X1, the distance between the feature vector of the object and the feature vectors of each object in the object set Sn is calculated; if there is a distance less than the second distance threshold among the distances between the feature vector of the object and the feature vectors of each object in the object set Sn, it indicates that there is a mis-identified object similar to the object in the object set Sn, which reflects that the object is also largely a mis-identified object, but was not identified in the first filtering. Therefore, the object is used as a mis-identified object under the target category. After the above two filtrations, the remaining structured data is basically the structured data of the objects under the target category, making the results of subsequent operations such as data search, image search by image, and trajectory tracking based on the structured data more accurate.

[0058] In the above embodiment, the first distance threshold and the second distance threshold are used to measure the similarity between the objects included in each structured data and the target object. The value selection method of the first distance threshold and the second distance threshold is not limited here, and can be specifically set according to the category of the target object. Moreover, the first distance threshold and the second distance threshold can be the same or different, and can be flexibly adjusted according to specific needs on the basis of ensuring the filtering accuracy.

[0059] In an embodiment of the present application, feature extraction can be performed based on the object category and the features of the object in the video data, structured data corresponding to the object category can be generated, and structured data under the target category can be identified therefrom. In view of the situation where there is misidentification during the recognition process, the embodiment of the present application can measure the similarity between the identified object and the target object by calculating the distance between the feature vector of the object contained in each structured data and the feature vector of the object under the target category, and filter out the structured data corresponding to the object that does not meet the similarity condition, thereby improving the accuracy of the recognition result. In addition, in order to reduce the impact of the misidentified objects missed during the filtering process on the structured data under the target category. It is also possible to perform secondary filtering on the structured data that has been filtered once based on the object set containing the misidentified object, further improving the accuracy of the target structured data, and providing an accurate data basis for subsequent operations such as search and trajectory tracking based on the target category structured data.

[0060] Regarding the above Figure 1a and Figure 1b System embodiment, the present application also provides a video processing method, Figure 3a FIG. 1 is a flow chart of the video processing method. Figure 3a As shown, the method includes:

[0061] S1a. Receive video data uploaded by a client, where the video data includes objects of at least one category.

[0062] S2a, performing structured processing on the video data to obtain structured data under at least one category, each structured data corresponds to an object under the category to which it belongs and includes a feature vector of the object.

[0063] S3a. For the target category, based on the feature vectors of the objects included in the structured data under the target category, the structured data corresponding to the misidentified objects under the target category are filtered out.

[0064] The target category is any category of at least one category, and the misidentified object refers to an object in the video data that is misidentified as a target category.

[0065] In the embodiments of the present application, the video data uploaded by the client contains at least one type of object. Therefore, at least one type of object in the video data can be classified and each type of object can be structured to generate structured data containing object feature vectors. Further, the structured data under the target category can be identified from the structured data under at least one category, and based on the feature vectors of the objects included in the structured data under the target category, the structured data of the mis-identified objects in the process of identifying the target category can be filtered out to improve the accuracy of the identification result. In an alternative embodiment, the unfiltered structured data can be output to a storage system for storage, providing a data basis for subsequent operations based on the structured data.

[0066] In view of the above Figure 1a and Figure 1b system embodiments, the present application also provides a data filtering method, Figure 3b which is a flowchart of the data filtering method. As Figure 3b shown, the method includes:

[0067] S1b. Receive the structured data under at least one category output by the video processing device. Each structured data corresponds to an object under its respective category and includes the feature vector of the object.

[0068] S2b. For the target category, based on the feature vectors of the objects included in the structured data under the target category, filter out the structured data corresponding to the mis-identified objects under the target category.

[0069] S3b. Output the unfiltered structured data under the target category to the storage system corresponding to the video processing device for storage.

[0070] Wherein, the target category is any one of the at least one category, and the mis-identified object refers to the object in the video data that is mis-identified as the target category.

[0071] In the embodiments of the present application, the structured data provided by the video processing device contains multiple types of objects and their feature vectors. Therefore, the structured data under the target category can be identified from the structured data. And based on the feature vectors of the objects included in the structured data under the target category, the structured data of the mis-identified objects in the process of identifying the target category can be filtered out to improve the accuracy of the identification result; and the unfiltered structured data can be stored in the storage system, providing a data basis for subsequent operations based on the structured data.

[0072] Optionally, in the above method embodiments, when filtering out the structured data of mis-identified objects corresponding to the target category based on the feature vectors of the objects included in the structured data under the target category, the central vector of the objects under the target category can be obtained from the feature vectors of the objects included in each structured data under the target category. This central vector can be used as a feature vector for measuring the similarity degree of each object to the objects under the target category. Furthermore, based on the central vector and the feature vectors of the objects included in each structured data, the mis-identified objects under the target category can be identified, and the structured data of the mis-identified objects corresponding to the target category can be filtered out.

[0073] In an alternative embodiment, to improve the accuracy of the recognition result, the central vector can also be optimized. When identifying the mis-identified objects under the target category based on the central vector and the feature vectors of the objects included in each structured data, the mean vector can be obtained by calculating the mean of the feature vectors of the objects included in each structured data. Further, according to the distances between the feature vectors of the objects included in each structured data and the mean vector, the feature vectors of the top N objects closest to the mean vector can be selected, and the mean of the feature vectors of the top N objects can be calculated as the optimized central vector of the objects under the target category. Here, N is an integer greater than or equal to 2; the distances between the feature vectors of the objects included in each structured data and the mean vector are used to measure the similarity degree of each object to the objects of the target category.

[0074] Furthermore, based on the optimized central vector and the feature vectors of the objects included in each structured data, the mis-identified objects under the target category can be identified. Optionally, the distances between the feature vectors of the objects included in each structured data and the optimized central vector can be calculated, and the objects with distances greater than the first distance threshold from the optimized central vector can be regarded as the mis-identified objects under the target category. In another alternative embodiment, to avoid the structured data after filtering out the mis-identified objects still including mis-identified objects, further result filtering can be performed on the structured data after filtering out the mis-identified objects. When calculating the distances between the feature vectors of the objects included in each structured data and the optimized central vector, for the objects with distances less than or equal to the first distance threshold from the optimized central vector, the distances between the feature vectors of the objects and the feature vectors of each object in the object set can be calculated; if there are distances less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, then the object is regarded as the mis-identified object under the target category; where the object set stores the mis-identified objects under the target category.

[0075] In the above embodiments, the first distance threshold and the second distance threshold are used to measure the similarity between the objects included in each structured data and the target object, and there is no limitation on the implementation manner. They can be specifically set according to the different categories of the target object. Moreover, the first distance threshold and the second distance threshold can be the same or different. On the basis of ensuring the filtering accuracy, they can be flexibly adjusted according to specific requirements.

[0076] In an alternative embodiment, when an erroneously recognized object is identified, the similarity between the erroneously recognized object and each object in the object set can also be calculated according to the feature vector of the erroneously recognized object; if the similarity between the erroneously recognized object and each object in the object set is less than the similarity threshold, the erroneously recognized object is added to the object set as the secondary filtering sample set for the erroneously recognized object in the above embodiments.

[0077] Figure 3c This is a schematic flowchart of another video processing method provided by an exemplary embodiment of the present application. As Figure 3c shown, it includes:

[0078] S1c. Upload the video data to a video processing device to request structured processing of the video data, where the video data includes at least one category of objects;

[0079] S2c. Receive the unfiltered structured data under at least one category returned by the video processing device and the structured data filtered according to the feature vectors of the objects included in the structured data;

[0080] S3c. Correspondingly display the unfiltered and filtered structured data under at least one category for the user to know the objects erroneously recognized under at least one category.

[0081] In this embodiment, the video data to be structured can be uploaded to a video processing device for structured processing. Among them, the video data to be structured can be the video data collected in real time at the local end or the video data collected and uploaded by other collection devices. The process of the video processing device performing structured processing on the video data includes two operations: obtaining structured data through structured processing and filtering the structured data. For the detailed implementation of these two operations, reference can be made to the foregoing embodiments and will not be elaborated here.

[0082] In this embodiment, after the video processing device performs structured processing on video data, at least one type of unfiltered structured data and filtered structured data can be obtained; among them, the filtered structured data is obtained by filtering the unfiltered structured data according to the feature vectors of the objects in the unfiltered structured data. Then, the video processing device will also return at least one type of unfiltered structured data and filtered structured data. After receiving the at least one type of unfiltered structured data and filtered structured data returned by the video processing device, the corresponding unfiltered structured data and filtered structured data of at least one type are displayed, which is convenient for the user to compare the structured data before and after filtering and intuitively understand the objects mis-identified in each category.

[0083] Figure 3d It is a schematic flowchart of an information query method provided by an exemplary embodiment of the present application. As Figure 3d shown, it includes:

[0084] S1d. Send a first query request to request to query information about a first object, and the first object belongs to the target category;

[0085] S2d. Receive the first information returned according to the first query request, and the first information is queried from the unfiltered structured data under the target category;

[0086] S3d. When it is determined that the first information does not belong to the first object, send a second query request to request to re-query information about the first object;

[0087] S4d. Receive the second information returned according to the second query request and display the second information as the information of the first object, and the second information is queried from the structured data filtered according to the feature vectors of the objects included in the structured data under the target category.

[0088] In this embodiment, information queries can be based on structured data. When a query is needed, a first query request can be sent, and the first query request can carry identification information pointing to the first object, such as the name, avatar, or other known information that can describe the first object. For example, if the first object is a pedestrian, in the case of not knowing the pedestrian's name and identity information, the first object can be described by information such as the pedestrian's gender and age, and queries can be made through the structured data, so as to obtain information such as the name and identity information of the pedestrian that is currently unknown. In addition, the first query request also includes the object category to which the first object belongs. For the convenience of description, in this embodiment, the object category to which the first object belongs is referred to as the target category.

[0089] In this embodiment, after obtaining the first query request, first query from the unfiltered structured data under the target category to obtain the first information, and return the first information. Among them, the unfiltered structured data under the target category corresponds to an object under the target category, and also includes the feature vector of the object. Among them, the video data containing the object under the target category can be pre-structured to obtain the unfiltered structured data under the target category. For the process of structured processing, reference can be made to the foregoing embodiments.

[0090] After obtaining the first information, it can be determined whether the first information belongs to the first object. Optionally, the matching degree between the first information and the first object can be calculated, and based on the matching degree, it can be determined whether the first information belongs to the first object; if the matching degree is high (for example, greater than the set matching degree threshold), it is considered that the first information belongs to the first object; otherwise, if the matching degree is low (for example, less than the set matching degree threshold), it is considered that the first information does not belong to the first object. In addition, the first information can also be output for the user to determine whether the first information belongs to the first object. Taking the first object as a certain pedestrian and querying the identity information of the pedestrian through the head portrait of the pedestrian as an example, the first information queried is an identity information, and the identity information represented by the first information is output to the user. If the user finds that the user portrait corresponding to the identity information is different from the head portrait of the pedestrian, for example, it can be determined that the first information does not belong to the first object.

[0091] In this case, a second query request can be sent again to request a re-query of the information of the first object. Since it is necessary to re-query the information of the first object, it can be determined that the accuracy of the unfiltered structured data under the target category may not be sufficient. In view of this, according to the second query request, query in the filtered structured data under the target category to obtain the second information. Optionally, according to the second query request, based on the feature vector of the object included in the unfiltered structured data under the target category, filter the unfiltered structured data under the target category to obtain the filtered structured data under the target category; then, query in the filtered structured data under the target category to obtain the second information. Or, during the structured processing process, the unfiltered structured data under the target category can be pre-filtered based on the feature vector of the object included in the unfiltered structured data under the target category to obtain the filtered structured data under the target category. Then, after receiving the second query request, the query can be directly performed in the pre-obtained filtered structured data under the target category to obtain the second information. Whether it is pre-filtering or real-time filtering, for the detailed implementation of the filtering process, reference can be made to the foregoing embodiments and will not be elaborated here.

[0092] After obtaining the second information, the second information can be returned. In this embodiment, since the second information is retrieved from the filtered structured data with relatively high precision and has a relatively high probability of being the information of the first object, the second information can be directly output as the information of the first object. Of course, before outputting the second information as the information of the first object, the matching degree between the second information and the first object can also be calculated, and it can be determined whether the second information belongs to the first object based on the matching degree, and when it is determined that the second information belongs to the first object, the second information is output as the information of the first object. Alternatively, after directly outputting the second information as the information of the first object, the user can further determine whether the second information belongs to the first object; if it is determined that the second information belongs to the first object, subsequent operations can be performed based on the second information; if it is determined that the second information does not belong to the first object, other methods can be used in a timely manner to obtain the information of the first object to ensure the success rate and timeliness of information acquisition.

[0093] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps S1a to S3a can be device A; for another example, the execution subject of steps S1a and S2a can be device A, and the execution subject of step S3a can be device B; and so on.

[0094] In addition, in some of the processes described in the above embodiments and the accompanying drawings, there are multiple operations that appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. The operation numbers such as S1a, S2a, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0095] Figure 4 This is a schematic structural diagram of a data processing device provided in another exemplary embodiment of the present application. As Figure 4 shown, the data processing device includes: a processor 41 and a memory 42 storing a computer program; wherein, the processor 41 and the memory 42 can be one or more.

[0096] The memory 42 is mainly used to store computer programs, which can be executed by the processor, so that the processor 41 controls the data processing device to realize corresponding functions, complete corresponding actions or tasks. In addition to storing computer programs, the memory can also be configured to store various other data to support operations on the data processing device, examples of which include instructions for any application or method operating on the data processing device.

[0097] The memory 42 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0098] In the embodiment of the present application, the implementation form of the processor 41 is not limited, for example, it can be but not limited to a CPU, a GPU or an MCU. The processor 41 can be regarded as a control system of a data processing device, which can be used to execute a computer program stored in the memory 42 to control the data processing device to implement corresponding functions and complete corresponding actions or tasks. It is worth noting that, depending on the implementation form of the data processing device and the different scenarios in which it is located, the functions to be implemented, the actions or tasks to be completed will be different; accordingly, the computer programs stored in the memory 42 will also be different, and the processor 41 can control the data processing device to implement different functions and complete different actions or tasks by executing different computer programs.

[0099] In an alternative embodiment, if Figure 4 As shown, the data processing device further includes other components such as a communication component 43 and a power supply component 44. Figure 4 The following only shows some components, which does not mean that the database engine only includes Figure 4 Components shown.

[0100] In an embodiment of the present application, when the processor 41 executes the computer program in the memory 42, it is used to: receive video data uploaded by a client, the video data including objects of at least one category; perform structured processing on the video data to obtain structured data under at least one category, each structured data corresponds to an object under the category to which it belongs, and includes a feature vector of the object; for a target category, based on the feature vector of the object included in the structured data under the target category, filter out the structured data corresponding to the misidentified object under the target category; wherein the target category is any category of at least one category, and the misidentified object refers to an object in the video data that is mistakenly identified as the target category.

[0101] In an alternative embodiment, the processor 41 is further configured to: output the unfiltered structured data under the target category to the storage system for storage.

[0102] In an alternative embodiment, when filtering the structured data of the mis-identified objects corresponding to the target category based on the feature vectors of the objects included in the structured data under the target category, the processor 41 is configured to: obtain the central vector of the objects under the target category from the feature vectors of the objects included in each structured data under the target category; identify the mis-identified objects under the target category according to the central vector and the feature vectors of the objects included in each structured data; and filter the structured data of the mis-identified objects corresponding to the target category.

[0103] In an alternative embodiment, when identifying the mis-identified objects under the target category according to the central vector and the feature vectors of the objects included in each structured data, the processor 41 is configured to: calculate the mean value of the feature vectors of the objects included in each structured data to obtain the mean vector; select the feature vectors of the top N objects closest to the mean vector according to the distances between the feature vectors of the objects included in each structured data and the mean vector; and calculate the mean value of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2.

[0104] In an alternative embodiment, when identifying the mis-identified objects under the target category according to the central vector and the feature vectors of the objects included in each structured data, the processor 41 is further configured to: calculate the distances between the feature vectors of the objects included in each structured data and the central vector, and use the objects with distances greater than the first distance threshold from the central vector as the mis-identified objects under the target category.

[0105] In an alternative embodiment, the processor 41 is further configured to: for the objects with distances less than or equal to the first distance threshold from the central vector, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, use the object as the mis-identified object under the target category; where the object set stores the mis-identified objects under the target category.

[0106] In an alternative embodiment, the processor 41 is further configured to: calculate the similarities between the mis-identified objects and each object in the object set according to the feature vectors of the mis-identified objects; and if the similarities between the mis-identified objects and each object in the object set are all less than the similarity threshold, add the mis-identified objects to the object set.

[0107] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement the steps executable by the data processing device in the above method embodiments.

[0108] The embodiment of the present application further provides a computer device, and the structure of this computer device is the same as or similar to that of Figure 4 the data processing device shown, which is not illustrated here. The computer device of this embodiment can be implemented as a video acquisition device such as a camera or an electronic eye, or can also be implemented as a terminal device such as a mobile phone or a computer. The computer device of this embodiment at least includes: a memory, a processor, and a communication component. Among them, the memory is used to store computer programs; the processor is used to execute the computer programs in the memory for: uploading video data to a video processing device to request structured processing of the video data, where the video data includes at least one category of objects; receiving unfiltered structured data under at least one category and structured data filtered according to the feature vectors of the objects included in the structured data returned by the video processing device; correspondingly displaying the unfiltered and filtered structured data under at least one category for the user to know the objects mis-identified under at least one category.

[0109] The embodiment of the present application further provides a computer device, and the structure of this computer device is the same as or similar to that of Figure 4 the data processing device shown, which is not illustrated here. The computer device of this embodiment can be implemented as a video acquisition device such as a camera or an electronic eye, or can also be implemented as a terminal device such as a mobile phone or a computer. The computer device of this embodiment at least includes: a memory, a processor, and a communication component. Among them, the memory is used to store computer programs; the processor is used to execute the computer programs in the memory for: sending a first query request to request querying information of a first object, where the first object belongs to a target category; receiving first information returned according to the first query request, and the first information is queried from the unfiltered structured data under the target category; in the case of determining that the first information does not belong to the first object, sending a second query request to request re-querying information of the first object; receiving second information returned according to the second query request and displaying the second information as the information of the first object, and the second information is queried from the structured data filtered according to the feature vectors of the objects included in the structured data under the target category.

[0110] The above Figure 4The communication component therein is configured to facilitate communication, either wired or wirelessly, between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0111] The above Figure 4 The power component therein provides power for various components of the device where the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device where the power component is located.

[0112] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce a device for realizing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0114] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device that realizes the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or multiple processes in the flowchart and / or one block or multiple blocks in the block diagram.

[0116] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0117] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0118] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0119] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0120] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the present application.

Claims

1. A video processing method, characterized in that, Including: Receiving video data uploaded by a client, where the video data includes at least one category of objects; Performing structured processing on the video data to obtain structured data under at least one category. Each structured data corresponds to an object under its respective category and includes a feature vector of the object; For a target category, calculating the mean value of the feature vectors of the objects included in the structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in the structured data and the mean vector, selecting the feature vectors of the top N objects closest to the mean vector; calculating the mean value of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculating the distances between the feature vectors of the objects included in the structured data and the central vector; taking the objects with distances greater than a first distance threshold from the central vector as mis-identified objects under the target category; for the objects with distances less than or equal to the first distance threshold from the central vector, calculating the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than a second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, taking the object as a mis-identified object under the target category; where the mis-identified objects under the target category are stored in the object set; filtering out the structured data corresponding to the mis-identified objects under the target category; Wherein, the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

2. The method according to claim 1, characterized in that, Further including at least one of the following operations: Outputting the unfiltered structured data under the target category to a storage system for storage; Outputting the unfiltered structured data under the target category and the structured data before filtering under the target category to a storage system for storage; Marking the structured data corresponding to the mis-identified objects under the target category and correspondingly outputting the structured data with and without marks under the target category.

3. The method according to claim 1, wherein Further including: Calculating the similarity between the mis-identified object and each object in the object set according to the feature vector of the mis-identified object; If the similarities between the mis-identified object and each object in the object set are all less than a similarity threshold, adding the mis-identified object to the object set.

4. A data filtering method, characterized in that, Including: Receiving structured data under at least one category output by a video processing device. Each structured data corresponds to an object under its respective category and includes a feature vector of the object; wherein, the structured data is obtained by performing structured processing on video data; For the target category, calculate the mean of the feature vectors of the objects included in each of the structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each of the structured data and the mean vector, select the feature vectors of the top N objects closest to the mean vector; calculate the mean of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each of the structured data and the central vector; use the objects with distances from the central vector greater than the first distance threshold as the mis-identified objects under the target category; for the objects with distances from the central vector less than or equal to the first distance threshold, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, use the object as the mis-identified object under the target category; where the mis-identified objects under the target category are stored in the object set; filter out the structured data corresponding to the mis-identified objects under the target category; Output the unfiltered structured data under the target category to the storage system corresponding to the video processing device for storage; Where the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

5. A video processing method, characterized in that, Includes: Receive the surveillance video in the specified environmental space uploaded by the video capture device, and the video capture device is installed in the specified environmental space; Perform structured processing on the surveillance video to obtain structured data under at least one object category, each structured data corresponds to an object under its respective object category and includes the feature vector of the object; For the target category, calculate the mean of the feature vectors of the objects included in each of the structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each of the structured data and the mean vector, select the feature vectors of the top N objects closest to the mean vector; calculate the mean of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each of the structured data and the central vector; use the objects with distances from the central vector greater than the first distance threshold as the mis-identified objects under the target category; for the objects with distances from the central vector less than or equal to the first distance threshold, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, use the object as the mis-identified object under the target category; where the mis-identified objects under the target category are stored in the object set; filter out the structured data corresponding to the mis-identified objects under the target category; Wherein, the target category is any one of the at least one object category, and the mis-identified object refers to an object in the surveillance video that is mis-identified as the target category.

6. The method according to claim 5, wherein The specified environmental space is the interior space of a public transportation vehicle, or an office park, or an urban road traffic space, or a self-service supermarket space, or a campus space.

7. A video processing method, characterized in that, Including: Uploading video data to a video processing device to request structured processing of the video data, where the video data includes at least one category of objects; Receiving unfiltered structured data under at least one category and structured data filtered according to the feature vectors of the objects included in the structured data returned by the video processing device; Correspondingly displaying the unfiltered and filtered structured data under at least one category for the user to know the mis-identified objects under at least one category; Wherein, the acquisition method of the structured data filtered according to the feature vectors of the objects included in the structured data is as follows: for the target category, calculate the mean value of the feature vectors of the objects included in each structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each structured data and the mean vector, select the first N object feature vectors closest to the mean vector; calculate the mean value of the first N object feature vectors as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each structured data and the central vector; take the objects with distances from the central vector greater than the first distance threshold as the mis-identified objects under the target category; for the objects with distances from the central vector less than or equal to the first distance threshold, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, take the object as the mis-identified object under the target category; wherein, the mis-identified objects under the target category are stored in the object set; filter out the structured data corresponding to the mis-identified objects under the target category.

8. An information query method, characterized in that, Including: Sending a first query request to request querying information of a first object, where the first object belongs to the target category; Receiving first information returned according to the first query request, where the first information is queried from the unfiltered structured data under the target category; When it is determined that the first information does not belong to the first object, sending a second query request to request re-querying information of the first object; Receiving second information returned according to the second query request and displaying the second information as the information of the first object, where the second information is queried from the structured data filtered according to the feature vectors of the objects included in the structured data under the target category; Among them, the method for obtaining the structured data filtered according to the feature vectors of the objects included in the structured data under the target category is as follows: calculate the mean value of the feature vectors of the objects included in each structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each structured data and the mean vector, select the feature vectors of the top N objects closest to the mean vector; calculate the mean value of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each structured data and the central vector; take the objects with distances greater than the first distance threshold from the central vector as the misidentified objects under the target category; for the objects with distances less than or equal to the first distance threshold from the central vector, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there are distances less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, take the object as the misidentified object under the target category; where the misidentified objects under the target category are stored in the object set; filter out the structured data corresponding to the misidentified objects under the target category.

9. A video processing device, characterized in that, Including: a processor and a memory storing a computer program; the processor is configured to execute the computer program for: receiving video data uploaded by a client, where the video data includes at least one category of objects; performing structured processing on the video data to obtain structured data under at least one category, each structured data corresponding to an object under its respective category and including the feature vector of the object; for a target category, calculate the mean value of the feature vectors of the objects included in each structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each structured data and the mean vector, select the feature vectors of the top N objects closest to the mean vector; calculate the mean value of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each structured data and the central vector; take the objects with distances greater than the first distance threshold from the central vector as the misidentified objects under the target category; for the objects with distances less than or equal to the first distance threshold from the central vector, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there are distances less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, take the object as the misidentified object under the target category; where the misidentified objects under the target category are stored in the object set; filter out the structured data corresponding to the misidentified objects under the target category; Wherein, the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

10. A data processing system, characterized in that, Comprising: A client, a video processing device, and a storage system; wherein, the client is used to provide video data, and the video processing device is used to perform structured processing on the video data provided by the client to generate structured data and store the structured data in the storage system; When performing structured processing on video data, the video processing device is used to: Receive the video data uploaded by the client, where the video data includes objects of at least one category; Perform structured processing on the video data to obtain structured data under at least one category, each structured data corresponding to an object under its respective category and including the feature vector of the object; For the target category, calculate the mean value of the feature vectors of the objects included in each structured data under the target category to obtain a mean vector; according to the distances between the feature vectors of the objects included in each structured data and the mean vector, select the feature vectors of the top N objects closest to the mean vector; calculate the mean value of the feature vectors of the top N objects as the central vector of the objects under the target category; where N is an integer greater than or equal to 2; calculate the distances between the feature vectors of the objects included in each structured data and the central vector; use the objects with distances greater than the first distance threshold from the central vector as the mis-identified objects under the target category; for the objects with distances less than or equal to the first distance threshold from the central vector, calculate the distances between the feature vectors of the objects and the feature vectors of each object in the object set; if there is a distance less than the second distance threshold among the distances between the feature vectors of the object and the feature vectors of each object in the object set, use the object as the mis-identified object under the target category; where the object set stores the mis-identified objects under the target category; filter out the structured data corresponding to the mis-identified objects under the target category; Wherein, the target category is any one of the at least one category, and the mis-identified object refers to an object in the video data that is mis-identified as the target category.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to implement the steps in the method according to any one of claims 1-8.

Images