Video object retrieval method and device, electronic equipment and storage medium

Abstract

The invention relates to artificial intelligence, and discloses a video object retrieval method and device, electronic equipment and a storage medium, which can retrieve target personnel in places such as hospitals. The method comprises the following steps: identifying a preset object in a monitoring video by using a detection model, and segmenting the identified preset object by using an instance segmentation model to obtain a plurality of object feature sets; obtaining text description features of the to-be-retrieved object, and screening a target object feature set from the plurality of object feature sets according to the text description features; when the target object feature set contains features meeting a preset filtering condition, selecting a corresponding information filtering model according to the features meeting the preset filtering condition; and performing information filtering processing on the target object feature set by using an information filtering model to obtain an information filtering result, and summarizing a plurality of video frames in the monitoring video corresponding to the information filtering result to obtain a retrieval result. According to the method, the video object retrieval efficiency in places such as hospitals can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a video object retrieval method, device, electronic equipment and storage medium. Background Art

[0002] In the field of digital medicine, doctors, nurses, or medical personnel and their accompanying family members come and go in different medical treatment sites every day. As the number of people coming and going increases, a large number of surveillance cameras have been widely used. For example, a large number of surveillance cameras are installed in hospitals and physical examination centers, so that when there are situations such as medical personnel getting lost or needing to find a target person, people can be retrieved through surveillance videos.

[0003] Existing video person retrieval methods usually compare a given target person image with a surveillance video frame, and retrieve the person based on the similarity between the target person image and the surveillance video frame. This method is limited by the accuracy of the target person image, and a large amount of surveillance video data will be generated in places with a large flow of people, such as hospitals, which leads to low efficiency in person retrieval. Therefore, it is urgent to propose a more efficient video person retrieval method that can be applied to places with a large flow of people, such as hospitals. Summary of the invention

[0004] The present invention provides a video object retrieval method, device, electronic equipment and storage medium, the main purpose of which is to improve the efficiency of video character retrieval in places with large personnel flow such as hospitals.

[0005] To achieve the above object, the present invention provides a video object retrieval method, comprising:

[0006] Using a pre-trained detection model to identify a preset object in the surveillance video, and using a pre-trained instance segmentation model to segment the preset object to obtain multiple object feature sets;

[0007] Acquire text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets according to the text description features;

[0008] When the target object feature set includes features that meet the preset filtering conditions, selecting a corresponding information filtering model according to the features that meet the preset filtering conditions;

[0009] The information filtering model is used to perform information filtering processing on the target object feature set to obtain an information filtering result, and multiple video frames in the monitoring video corresponding to the information filtering result are summarized to obtain a retrieval result of the object to be retrieved.

[0010] Optionally, the filtering of the target object feature set from the plurality of object feature sets according to the text description feature includes:

[0011] Classifying different object features in a plurality of object feature sets to obtain feature categories corresponding to the object feature sets;

[0012] The object feature set corresponding to the feature category consistent with the text description feature is used as the target object feature set.

[0013] Optionally, the pre-trained instance segmentation model is used to segment the preset object to obtain multiple object feature sets, including:

[0014] Using the convolutional neural network in the instance segmentation model to extract features of the preset object to obtain multi-layer image features;

[0015] Performing feature fusion processing on the multi-layer image features to obtain fused image features;

[0016] Inputting the fused image features into the region proposal network in the instance segmentation model to perform region detection, and obtaining a plurality of human body proposal regions;

[0017] The features included in different human body proposal regions among the multiple human body proposal regions are used as multiple object feature sets.

[0018] Optionally, the selecting a corresponding information filtering model according to the features meeting the preset filtering conditions includes:

[0019] Selecting a preset filtering condition corresponding to the target object feature set;

[0020] Turning on a filter switch corresponding to the preset filtering condition;

[0021] In the model reference library, select the classification model corresponding to the enabled filter as the information filtering model.

[0022] Optionally, after obtaining the search results, the method further includes:

[0023] The search results are uploaded to a cloud server.

[0024] Optionally, before obtaining the text description features of the object to be retrieved, the method further includes:

[0025] Receiving a search request and determining a file format of the search request;

[0026] When the search request is in text format, extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0027] When the search request is in voice format, converting the search request in voice format into text format, and extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0028] When the search request is in an image format, the search request in the image format is converted into a text format, and a pre-trained keyword extraction model is used to extract description features in the text to obtain text description features.

[0029] Optionally, before using the pre-trained detection model to identify the preset object in the surveillance video, the method further includes:

[0030] Acquire an original data stream corresponding to a reference object, and extract a key point data set from the original data stream;

[0031] Identify the three-dimensional coordinate values ​​corresponding to the key point data set, and perform detection training processing on the deep neural network model according to the three-dimensional coordinate values ​​to obtain a trained detection model.

[0032] In order to solve the above problem, the present invention further provides a video object retrieval device, the device comprising:

[0033] An object segmentation module is used to identify a preset object in the surveillance video using a pre-trained detection model, and to segment the preset object using a pre-trained instance segmentation model to obtain multiple object feature sets;

[0034] A feature screening module, used for obtaining text description features of the object to be retrieved, and screening a target object feature set from the plurality of object feature sets according to the text description features;

[0035] A model selection module, used for selecting a corresponding information filtering model according to the features meeting the preset filtering conditions when the target object feature set includes features meeting the preset filtering conditions;

[0036] The result generation module is used to use the information filtering model to perform information filtering processing on the target object feature set to obtain information filtering results, and summarize multiple video frames in the monitoring video corresponding to the information filtering results to obtain the retrieval results of the object to be retrieved.

[0037] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video object retrieval method described above.

[0041] In order to solve the above problem, the present invention further provides a storage medium, wherein at least one computer program is stored in the storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned video object retrieval method.

[0042] In the embodiment of the present invention, accurate retrieval is performed based on text description features, thereby avoiding fuzzy retrieval when image information is incomplete. In addition, the solution of separating human body detection and human body segmentation from text description features can effectively realize modular design, easy deployment and strong usability. Therefore, the video object retrieval method, device, electronic device and storage medium proposed in the present invention can solve the problem of low efficiency of video character retrieval in places with large personnel flow such as hospitals. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A system architecture diagram of a video object retrieval method provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a flow chart of a video object retrieval method provided by an embodiment of the present invention;

[0045] Figure 3 A functional module diagram of a video object retrieval device provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the structure of an electronic device for implementing the video object retrieval method provided by an embodiment of the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0049] The embodiment of the present application provides a video object retrieval method. The execution subject of the video object retrieval method includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the video object retrieval method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0050] As one of the application scenarios, Figure 1 As shown, the present application can be applied in a server cluster consisting of a cloud server 1 and multiple edge servers 2. For example, in a medical scenario, the cloud server 1 can be located in a medical service center, and the edge server 2 can be located in various areas of the hospital. The edge server 2 is connected to the video surveillance device 2 for communication, receives the surveillance video captured by the video surveillance device 2, and stores the surveillance video in a local database. The video object retrieval method of the present invention mainly runs in the edge server 2.

[0051] In specific applications, the user can send a search request for a video object to the cloud server 1, and the cloud server 1 parses the search request to obtain text description features of the object to be searched, and sends the text description features to each edge server 2. The edge server 2 uses the video object search method described in the present application to filter out videos or video frames that meet the text description features from the local database, obtain search results, and send the search results to the cloud server 1 for further confirmation.

[0052] Further, refer to Figure 2 FIG. 1 is a flow chart of a video object retrieval method provided by an embodiment of the present invention. In this embodiment, the video object retrieval method includes the following steps S1-S4:

[0053] S1. Using a pre-trained detection model to identify a preset object in a surveillance video, and using a pre-trained instance segmentation model to segment the preset object to obtain a plurality of object feature sets.

[0054] In the embodiment of the present invention, since there are different video frames in the monitoring retrieval video, and not every frame in the different video frames necessarily contains the preset object, it is necessary to use a pre-trained detection model to identify the preset object in the monitoring video.

[0055] In this solution, the preset object may be a human object. In other application scenarios, the preset object may also be an object in other forms, such as an animal or a specified object. In one embodiment of the present invention, the preset object may be a person.

[0056] Specifically, before using the pre-trained detection model to identify the preset object in the surveillance video, the method further includes:

[0057] Acquire an original data stream corresponding to a reference object, and extract a key point data set from the original data stream;

[0058] Identify the three-dimensional coordinate values ​​corresponding to the key point data set, and perform detection training processing on the deep neural network model according to the three-dimensional coordinate values ​​to obtain a trained detection model.

[0059] In detail, when the detection model is a human body detection model, the reference object may be a human body, and the key point data set may be a skeleton joint point data set.

[0060] In one embodiment of the present invention, the original data stream of the reference actor can be obtained through the sensor, and the relevant data corresponding to the selected actor can be obtained by using the original data stream as a reference. The sensor can be a kinect sensor, and the skeleton key point data set in the original data stream can be extracted by coordinate transformation and smoothing filtering, and displayed by visual processing technology. The human body displayed by visual processing technology mainly displays the skeleton and the joints of the skeleton.

[0061] Preferably, the trained detection model can quickly detect the target object in the video, so as to filter out the video frames containing the target object from the surveillance video, so as to reduce the subsequent computational workload.

[0062] Furthermore, the pre-trained instance segmentation model is used to segment the preset object to obtain multiple object feature sets, including:

[0063] Using the convolutional neural network in the instance segmentation model to extract features of the preset object to obtain multi-layer image features;

[0064] Performing feature fusion processing on the multi-layer image features to obtain fused image features;

[0065] Inputting the fused image features into the region proposal network in the instance segmentation model to perform region detection, and obtaining a plurality of human body proposal regions;

[0066] The features included in different human body proposal regions among the multiple human body proposal regions are used as multiple object feature sets.

[0067] In detail, the instance segmentation model can be a MaskRCNN (Mask Region-based Convolutional Neural Network target detection and segmentation network) model. The Mask RCNN model is a machine learning model that realizes target detection and segmentation. It is an extension of FasterRCNN, and adds a branch on the basis of FasterRCNN to predict the detailed segmentation mask of each target.

[0068] In one embodiment of the present invention, when the preset object is a human body, the object feature set may include, but is not limited to: gender features, clothing features, height features, hairstyle features, body shape features, etc.

[0069] S2. Obtain text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets according to the text description features.

[0070] In the embodiment of the present invention, the text description feature refers to the text description of the object to be searched by the user to show the appearance features of the object to be searched. For example, if the object to be searched is a person, the text description feature can be: male, wearing patient clothes, etc.

[0071] Specifically, before obtaining the text description features of the object to be retrieved, the method may further include:

[0072] Receiving a search request and determining a file format of the search request;

[0073] When the search request is in text format, extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0074] When the search request is in voice format, converting the search request in voice format into text format, and extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0075] When the search request is in an image format, the search request in the image format is converted into a text format, and a pre-trained keyword extraction model is used to extract description features in the text to obtain text description features.

[0076] Preferably, in this solution, obtaining the text description features of the object to be searched may be receiving a user's voice request. For example, if a user wants to find a man in patient clothes who is lost in a hospital, the user's semantic search request may be "find a man in patient clothes". The text description features corresponding to the user's semantic search request are: man, patient clothes.

[0077] Furthermore, the extracting description features in the text using a pre-trained keyword extraction model to obtain text description features includes: determining whether there is request data in the text that is consistent with a reference parsing category;

[0078] The request data consistent with the reference parsing category is used as a text description feature. In detail, the multiple reference parsing categories can be determined based on the object feature set obtained by segmenting the preset object, such as gender, clothing, height, hairstyle, body shape, etc.

[0079] Furthermore, the embodiment of the present invention uses the object feature set corresponding to the feature category consistent with the text description feature as the target object feature set.

[0080] S3. When the target object feature set includes features that meet the preset filtering conditions, a corresponding information filtering model is selected according to the features that meet the preset filtering conditions.

[0081] In the embodiment of the present invention, the selecting a corresponding information filtering model according to the features meeting the preset filtering conditions includes:

[0082] Selecting a preset filtering condition corresponding to the target object feature set;

[0083] Turning on a filter switch corresponding to the preset filtering condition;

[0084] In the model reference library, select the classification model corresponding to the enabled filter as the information filtering model.

[0085] In detail, different classification models have corresponding filter switches. For example, the classification model can be a height classification module, a clothing classification module, or a gender classification module, etc. Only when the target object feature set includes a height feature, the filter switch corresponding to the height classification model will be turned on. The classification model is a trained model based on DenseNet and other methods for classifying a single feature of the human body.

[0086] S4. Perform information filtering processing on the target object feature set using the information filtering model to obtain an information filtering result, and summarize multiple video frames in the surveillance video corresponding to the information filtering result to obtain a retrieval result of the object to be retrieved.

[0087] In the embodiment of the present invention, the target object feature set is input into the information filtering model to perform information filtering processing to obtain an information filtering result.

[0088] Furthermore, the embodiment of the present invention can upload the search results to a cloud server.

[0089] In the embodiment of the present invention, accurate retrieval is performed based on text description features, thereby avoiding the occurrence of fuzzy retrieval when image information is incomplete. In addition, the solution of separating human body detection and human body segmentation from text description features can effectively realize modular design, easy deployment and strong usability. Therefore, the video object retrieval method proposed by the present invention can solve the problem of low efficiency of video character retrieval.

[0090] like Figure 3 , which is a functional module diagram of a video object retrieval device provided by an embodiment of the present invention.

[0091] The video object retrieval device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the video object retrieval device 100 can include an object segmentation module 101, a feature screening module 102, a model selection module 103 and a result generation module 104. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, which are stored in the memory of the electronic device.

[0092] In this embodiment, the functions of each module / unit are as follows:

[0093] The object segmentation module 101 is used to identify a preset object in the surveillance video using a pre-trained detection model, and to segment the preset object using a pre-trained instance segmentation model to obtain a plurality of object feature sets;

[0094] The feature screening module 102 is used to obtain text description features of the object to be searched, and screen a target object feature set from the multiple object feature sets according to the text description features;

[0095] The model selection module 103 is used to select a corresponding information filtering model according to the features that meet the preset filtering conditions when the target object feature set includes features that meet the preset filtering conditions;

[0096] The result generation module 104 is used to perform information filtering processing on the target object feature set using the information filtering model to obtain an information filtering result, and to summarize multiple video frames in the surveillance video corresponding to the information filtering result to obtain a retrieval result of the object to be retrieved.

[0097] In detail, the specific implementation of each module of the video object retrieval device 100 is as follows:

[0098] Step 1: Use a pre-trained detection model to identify a preset object in the surveillance video, and use a pre-trained instance segmentation model to segment the preset object to obtain multiple object feature sets.

[0099] In the embodiment of the present invention, since there are different video frames in the monitoring retrieval video, and not every frame in the different video frames necessarily contains the preset object, it is necessary to use a pre-trained detection model to identify the preset object in the monitoring video.

[0100] In this solution, the preset object may be a human object. In other application scenarios, the preset object may also be an object in other forms, such as an animal or a specified object. In one embodiment of the present invention, the preset object may be a person.

[0101] Specifically, before using the pre-trained detection model to identify the preset object in the surveillance video, the method further includes:

[0102] Acquire an original data stream corresponding to a reference object, and extract a key point data set from the original data stream;

[0103] Identify the three-dimensional coordinate values ​​corresponding to the key point data set, and perform detection training processing on the deep neural network model according to the three-dimensional coordinate values ​​to obtain a trained detection model.

[0104] In detail, when the detection model is a human body detection model, the reference object may be a human body, and the key point data set may be a skeleton joint point data set.

[0105] In one embodiment of the present invention, the original data stream of the reference actor can be obtained through the sensor, and the relevant data corresponding to the selected actor can be obtained by using the original data stream as a reference. The sensor can be a kinect sensor, and the skeleton key point data set in the original data stream can be extracted by coordinate transformation and smoothing filtering, and displayed by visual processing technology. The human body displayed by visual processing technology mainly displays the skeleton and the joints of the skeleton.

[0106] Preferably, the trained detection model can quickly detect the target object in the video, so as to filter out the video frames containing the target object from the surveillance video, so as to reduce the subsequent computational workload.

[0107] Furthermore, the pre-trained instance segmentation model is used to segment the preset object to obtain multiple object feature sets, including:

[0108] Using the convolutional neural network in the instance segmentation model to extract features of the preset object to obtain multi-layer image features;

[0109] Performing feature fusion processing on the multi-layer image features to obtain fused image features;

[0110] Inputting the fused image features into the region proposal network in the instance segmentation model to perform region detection, and obtaining a plurality of human body proposal regions;

[0111] The features included in different human body proposal regions among the multiple human body proposal regions are used as multiple object feature sets.

[0112] In detail, the instance segmentation model can be a MaskRCNN (Mask Region-based Convolutional Neural Network target detection and segmentation network) model. The Mask RCNN model is a machine learning model that realizes target detection and segmentation. It is an extension of FasterRCNN, and adds a branch on the basis of FasterRCNN to predict the detailed segmentation mask of each target.

[0113] In one embodiment of the present invention, when the preset object is a human body, the object feature set may include, but is not limited to: gender features, clothing features, height features, hairstyle features, body shape features, etc.

[0114] Step 2: Obtain text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets based on the text description features.

[0115] In the embodiment of the present invention, the text description feature refers to the text description of the object to be searched by the user to show the appearance features of the object to be searched. For example, if the object to be searched is a person, the text description feature can be: male, wearing patient clothes, etc.

[0116] Specifically, before obtaining the text description features of the object to be retrieved, the method may further include:

[0117] Receiving a search request and determining a file format of the search request;

[0118] When the search request is in text format, extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0119] When the search request is in voice format, converting the search request in voice format into text format, and extracting description features in the text using a pre-trained keyword extraction model to obtain text description features;

[0120] When the search request is in an image format, the search request in the image format is converted into a text format, and a pre-trained keyword extraction model is used to extract description features in the text to obtain text description features.

[0121] Preferably, in this solution, obtaining the text description features of the object to be searched may be receiving a user's voice request. For example, if a user wants to find a man in patient clothes who is lost in a hospital, the user's semantic search request may be "find a man in patient clothes". The text description features corresponding to the user's semantic search request are: man, patient clothes.

[0122] Furthermore, the extracting description features in the text using a pre-trained keyword extraction model to obtain text description features includes: determining whether there is request data in the text that is consistent with a reference parsing category;

[0123] The request data consistent with the reference parsing category is used as a text description feature. In detail, the multiple reference parsing categories can be determined based on the object feature set obtained by segmenting the preset object, such as gender, clothing, height, hairstyle, body shape, etc.

[0124] Furthermore, the embodiment of the present invention uses the object feature set corresponding to the feature category consistent with the text description feature as the target object feature set.

[0125] Step three: when the target object feature set includes features that meet the preset filtering conditions, select a corresponding information filtering model according to the features that meet the preset filtering conditions.

[0126] In the embodiment of the present invention, the selecting a corresponding information filtering model according to the features meeting the preset filtering conditions includes:

[0127] Selecting a preset filtering condition corresponding to the target object feature set;

[0128] Turning on a filter switch corresponding to the preset filtering condition;

[0129] In the model reference library, select the classification model corresponding to the enabled filter as the information filtering model.

[0130] In detail, different classification models have corresponding filter switches. For example, the classification model can be a height classification module, a clothing classification module, or a gender classification module, etc. Only when the target object feature set includes a height feature, the filter switch corresponding to the height classification model will be turned on. The classification model is a trained model based on DenseNet and other methods for classifying a single feature of the human body.

[0131] Step 4: Use the information filtering model to perform information filtering processing on the target object feature set to obtain an information filtering result, and summarize multiple video frames in the surveillance video corresponding to the information filtering result to obtain a retrieval result of the object to be retrieved.

[0132] In the embodiment of the present invention, the target object feature set is input into the information filtering model to perform information filtering processing to obtain an information filtering result.

[0133] Furthermore, the embodiment of the present invention can upload the search results to a cloud server.

[0134] In the embodiment of the present invention, accurate retrieval is performed based on text description features, thereby avoiding the occurrence of fuzzy retrieval when image information is incomplete. In addition, the solution of separating human body detection and human body segmentation from text description features can effectively realize modular design, easy deployment and strong usability. Therefore, the video object retrieval device proposed by the present invention can solve the problem of low efficiency of video character retrieval.

[0135] like Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a video object retrieval method provided by an embodiment of the present invention.

[0136] In the embodiment of the present invention, the electronic device may be Figure 1 The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a video object retrieval program.

[0137] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device by running or executing programs or modules stored in the memory 11 (for example, executing video object retrieval programs, etc.), and calling data stored in the memory 11.

[0138] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device, such as the code of a video object retrieval program, etc., but may also be used to temporarily store data that has been output or is to be output.

[0139] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0140] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0141] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0142] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0143] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0144] The video object retrieval program stored in the memory 11 of the electronic device is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0145] Using a pre-trained detection model to identify a preset object in the surveillance video, and using a pre-trained instance segmentation model to segment the preset object to obtain multiple object feature sets;

[0146] Acquire text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets according to the text description features;

[0147] When the target object feature set includes features that meet the preset filtering conditions, selecting a corresponding information filtering model according to the features that meet the preset filtering conditions;

[0148] The information filtering model is used to perform information filtering processing on the target object feature set to obtain an information filtering result, and multiple video frames in the monitoring video corresponding to the information filtering result are summarized to obtain a retrieval result.

[0149] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0150] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. The storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0151] The present invention further provides a storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0152] Using a pre-trained detection model to identify a preset object in the surveillance video, and using a pre-trained instance segmentation model to segment the preset object to obtain multiple object feature sets;

[0153] Acquire text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets according to the text description features;

[0154] When the target object feature set includes features that meet the preset filtering conditions, selecting a corresponding information filtering model according to the features that meet the preset filtering conditions;

[0155] The information filtering model is used to perform information filtering processing on the target object feature set to obtain an information filtering result, and multiple video frames in the monitoring video corresponding to the information filtering result are summarized to obtain a retrieval result.

[0156] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0157] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0159] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0160] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0161] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0162] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0163] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A video object retrieval method, characterized in that: The method comprises: Using a pre-trained detection model to identify a preset object in the surveillance video, and using a pre-trained instance segmentation model to segment the preset object to obtain multiple object feature sets; Acquire text description features of the object to be retrieved, and filter a target object feature set from the multiple object feature sets according to the text description features; When the target object feature set includes features that meet the preset filtering conditions, selecting a corresponding information filtering model according to the features that meet the preset filtering conditions; The information filtering model is used to perform information filtering processing on the target object feature set to obtain an information filtering result, and multiple video frames in the monitoring video corresponding to the information filtering result are summarized to obtain a retrieval result of the object to be retrieved.

2. The video object retrieval method according to claim 1, characterized in that: The step of selecting a target object feature set from the plurality of object feature sets according to the text description feature comprises: Classifying different object features in a plurality of object feature sets to obtain feature categories corresponding to the object feature sets; The object feature set corresponding to the feature category consistent with the text description feature is used as the target object feature set.

3. The video object retrieval method according to claim 1, characterized in that: The pre-trained instance segmentation model is used to segment the preset object to obtain multiple object feature sets, including: Using the convolutional neural network in the instance segmentation model to extract features of the preset object to obtain multi-layer image features; Performing feature fusion processing on the multi-layer image features to obtain fused image features; Inputting the fused image features into the region proposal network in the instance segmentation model to perform region detection, and obtaining a plurality of human body proposal regions; The features included in different human body proposal regions among the multiple human body proposal regions are used as multiple object feature sets.

4. The video object retrieval method according to claim 1, characterized in that: The selecting a corresponding information filtering model according to the features meeting the preset filtering conditions includes: Selecting a preset filtering condition corresponding to the target object feature set; Turning on a filter switch corresponding to the preset filtering condition; In the model reference library, select the classification model corresponding to the enabled filter as the information filtering model.

5. The video object retrieval method according to claim 1, characterized in that: After obtaining the search result of the object to be searched, the method further includes: The search results are uploaded to a cloud server.

6. The video object retrieval method according to claim 1, characterized in that: Before obtaining the text description features of the object to be retrieved, the method further includes: Receiving a search request and determining a file format of the search request; When the search request is in text format, extracting description features in the text using a pre-trained keyword extraction model to obtain text description features; When the search request is in voice format, converting the search request in voice format into text format, and extracting description features in the text using a pre-trained keyword extraction model to obtain text description features; When the search request is in an image format, the search request in the image format is converted into a text format, and a pre-trained keyword extraction model is used to extract description features in the text to obtain text description features.

7. The video object retrieval method according to claim 1, characterized in that: Before using the pre-trained detection model to identify the preset object in the surveillance video, the method further includes: Acquire an original data stream corresponding to a reference object, and extract a key point data set from the original data stream; Identify the three-dimensional coordinate values ​​corresponding to the key point data set, and perform detection training processing on the deep neural network model according to the three-dimensional coordinate values ​​to obtain a trained detection model.

8. A video object retrieval device, characterized in that: The device comprises: An object segmentation module is used to identify a preset object in the surveillance video using a pre-trained detection model, and to segment the preset object using a pre-trained instance segmentation model to obtain multiple object feature sets; A feature screening module, used for obtaining text description features of the object to be retrieved, and screening a target object feature set from the plurality of object feature sets according to the text description features; A model selection module, used for selecting a corresponding information filtering model according to the features meeting the preset filtering conditions when the target object feature set includes features meeting the preset filtering conditions; The result generation module is used to use the information filtering model to perform information filtering processing on the target object feature set to obtain information filtering results, and summarize multiple video frames in the monitoring video corresponding to the information filtering results to obtain the retrieval results of the object to be retrieved.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video object retrieval method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video object retrieval method according to any one of claims 1 to 7 is implemented.

Images