Video background object recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video background object recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing and artificial intelligence. The method comprises the following steps: acquiring current video stream data containing a background image; based on the current video stream data, performing object recognition on the background image to obtain video modal object category data; performing video frame extraction on the current video stream data to obtain video frame data; performing image segmentation on the video frame data to obtain image modal object category data; performing text description on the background image based on the video frame data to obtain a background description text; performing named entity recognition on the background description text to obtain text modal object category data; and performing background object category identification on the current video stream data based on the video modal object category data, the image modal object category data and the text modal object category data. According to the embodiment of the invention, the recognition accuracy of the video background object can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing and artificial intelligence technology, and in particular to a method and device for recognizing background objects in a video, an electronic device, and a storage medium. Background Art

[0002] Video background object recognition refers to the type identification of objects in the background image contained in the video data. For example, in the financial field scenario, by establishing a video connection with the user, the background image in the connection video data can be obtained. By performing video background object recognition on the objects in the background image, the category of the background object can be obtained. By analyzing the background object, the user's customer image can be enriched.

[0003] At present, the common method for video background object recognition is usually to manually identify objects in the video background during the video process. However, this method is inefficient and too subjective, and it is easy to cause errors in the recognition of video background objects. Therefore, how to improve the recognition accuracy of video background objects has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a face mask detection method and device, an electronic device and a storage medium, aiming to improve the recognition accuracy of video background objects.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for identifying background objects in a video, the method comprising:

[0006] Get the current video stream data including the background image;

[0007] Based on the current video stream data, performing object recognition on the background image to obtain video modality object category data;

[0008] Performing video frame extraction on the current video stream data to obtain video frame data;

[0009] Performing image segmentation on the video frame data to obtain image modality object category data;

[0010] Based on the video frame data, a text description is performed on the background image to obtain a background description text;

[0011] Performing named entity recognition on the background description text to obtain text modality object category data;

[0012] Based on the video modality object category data, the image modality object category data and the text modality object category data, background object category recognition is performed on the current video stream data.

[0013] In some embodiments, performing object recognition on the background image based on the current video stream data to obtain video modality object category data includes:

[0014] Performing frame extraction on the current video stream data to obtain an initial video frame sequence;

[0015] Performing image preprocessing on the initial video frame sequence to obtain a standard video frame sequence;

[0016] Extracting features from the standard video frame sequence to obtain a feature graph sequence;

[0017] Performing attention weight allocation on the feature map sequence to obtain feature map attention weight;

[0018] Based on the feature map attention weights, performing object detection on the feature map sequence to obtain an object bounding box;

[0019] Based on the object bounding box, the video modality object category data is generated.

[0020] In some embodiments, performing image segmentation on the video frame data to obtain image modality object category data includes:

[0021] Extracting features from the video frame data to obtain a video frame feature vector;

[0022] Performing instance recognition on the video frame feature vector to obtain a background instance;

[0023] Performing mask processing on the background instance to obtain an instance segmentation mask;

[0024] Information is extracted from the instance segmentation mask to obtain the image modality object category data.

[0025] In some embodiments, the step of performing a text description on the background image based on the video frame data to obtain a background description text includes:

[0026] Performing global feature extraction on the video frame data to obtain background features;

[0027] Extracting local features from the video frame data to obtain instance features;

[0028] Based on the background features and the instance features, performing context semantic understanding on the video frame data to obtain video frame context information;

[0029] Based on the video frame context information, performing image region weighting on the video frame data to obtain an instance weight;

[0030] The background description text is generated based on the background features, the instance features, the video frame context information and the instance weight.

[0031] In some embodiments, performing named entity recognition on the background description text to obtain text modality object category data includes:

[0032] Identify named entities in the background description text to obtain a candidate entity sequence;

[0033] Based on a preset background object library, the candidate entity sequence is screened to obtain the text modality object category data.

[0034] In some embodiments, the performing background object recognition on the current video stream data based on the video modality object category data, the image modality object category data, and the text modality object category data includes:

[0035] Performing image feature extraction on the video modality object category data to obtain a video modality object feature vector;

[0036] Performing image feature extraction on the image modality object category data to obtain an image modality object feature vector;

[0037] Performing text feature extraction on the text modality object category data to obtain a text modality object feature vector;

[0038] Feature fusion is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a background image object category.

[0039] In some embodiments, the step of performing feature fusion on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a background image object category includes:

[0040] Performing weight allocation on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain feature weight proportion data;

[0041] Based on the feature weight proportion data, a weighted sum operation is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a target feature vector;

[0042] Based on the target feature vector, the background image object category is determined.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video background object recognition device, the device comprising:

[0044] A video data acquisition module is used to acquire current video stream data including a background image;

[0045] A first object acquisition module, configured to perform object recognition on the background image based on the current video stream data to obtain video modality object category data;

[0046] A video frame extraction module, used for extracting video frames from the current video stream data to obtain video frame data;

[0047] A second object acquisition module, based on performing image segmentation on the video frame data, obtains image modality object category data;

[0048] A background object description module, used to perform text description on the background image based on the video frame data to obtain background description text;

[0049] A third object acquisition module is used to perform named entity recognition on the background description text to obtain text modality object category data;

[0050] The object feature fusion module is used to perform background object recognition on the current video stream data based on the video modality object category data, the image modality object category data and the text modality object category data.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0053] The video background object recognition method and device, electronic device and storage medium proposed in the present application perform object recognition on the background image in the current video stream data based on the acquired current video stream data to obtain video modal object category data, then perform image segmentation on the frequency frame data of the current video stream data to obtain image modal object category data, finally, perform text description on the background image based on the video frame data to obtain background description text, and perform named entity recognition on the background description text to obtain text modal object category data, thereby reducing the manual participation in video background object recognition and realizing multimodal recognition of the categories of objects contained in the background image, and then perform background object category recognition on the current video stream data based on the video modal object category data, the image modal object category data and the text modal object category data, thereby integrating the multimodal object type recognition results and improving the recognition accuracy of objects in the video background. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a method for identifying video background objects provided by an embodiment of the present application;

[0055] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0056] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0057] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0058] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0059] Figure 6 yes Figure 1 Flow chart of step S107 in FIG.

[0060] Figure 7 yes Figure 6 Flow chart of step S604 in FIG.

[0061] Figure 8 is a structural diagram of a video background object recognition device provided in an embodiment of the present application;

[0062] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Multimodality: Multimodality refers to combining and processing data from different sources or types to obtain more comprehensive information and understanding. By simultaneously analyzing text, images, audio, and other forms of data, multimodal technology can better capture and represent complex phenomena and is widely used in many fields such as computer vision, natural language processing, and human-computer interaction. Multimodal learning aims to utilize the mutual complementarity between modalities to improve the performance and generalization ability of the model, thereby achieving a more intelligent system.

[0068] Attention mechanism: The attention mechanism is a computational method that mimics human visual attention, which aims to enable the model to dynamically focus on the most relevant information when processing input data, thereby improving performance. Its benefit is that it can effectively capture long-distance dependencies and important features, significantly improving the accuracy and efficiency of tasks such as natural language processing, image recognition, and translation. By adaptively adjusting the focus of attention, the attention mechanism enhances the model's expressiveness and enables it to perform better in complex tasks.

[0069] Video background object recognition refers to the type identification of objects in the background image contained in the video data. For example, in the financial field scenario, by establishing a video connection with the user, the background image in the connection video data can be obtained. By performing video background object recognition on the objects in the background image, the category of the background object can be obtained. By analyzing the background object, the user's customer image can be enriched.

[0070] At present, the common method of video background object recognition is usually to manually identify objects in the video background during the video process. However, this method is inefficient and too subjective, and it is easy to cause errors in video background object recognition. Therefore, how to improve the accuracy of video background object recognition has become a technical problem that needs to be solved urgently.

[0071] Based on this, the embodiments of the present application provide a method and device for identifying background objects in a video, an electronic device, and a storage medium, aiming to improve the recognition accuracy of background objects in a video.

[0072] The video background object recognition method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video background object recognition method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The video background object recognition method provided in the embodiment of the present application relates to the field of image processing and artificial intelligence technology. The video background object recognition method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video background object recognition method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] Figure 1 is an optional flow chart of the video background object recognition method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0078] Step S101, obtaining current video stream data including a background image;

[0079] Step S102, based on the current video stream data, performing object recognition on the background image to obtain video modality object category data;

[0080] Step S103, extracting frames from the current video stream data to obtain video frame data;

[0081] Step S104, performing image segmentation on the video frame data to obtain image modality object category data;

[0082] Step S105, based on the video frame data, a text description is performed on the background image to obtain a background description text;

[0083] Step S106, performing named entity recognition on the background description text to obtain text modality object category data;

[0084] Step S107, based on the video modality object category data, the image modality object category data and the text modality object category data, background object category recognition is performed on the current video stream data.

[0085] Steps S101 to S107 shown in the embodiment of the present application obtain current video stream data including a background image, and perform object recognition on the background image based on the current video stream data to obtain video modal object category data. Further, perform video frame extraction on the above-mentioned current video stream data to obtain video frame data, and perform image segmentation on the video frame data to obtain image modal object category data. Secondly, perform text description on the background image based on the video frame data to obtain background description text, and perform named entity recognition on the background description text to obtain text modal object category data. Finally, perform background object category recognition on the current video stream data based on the above-mentioned video modal object category data, the above-mentioned image modal object category data and the above-mentioned text modal object category data. Therefore, the present application performs object recognition on the background image in the current video stream data based on the acquired current video stream data to obtain video modal object category data, and then performs image segmentation on the frame data of the current video stream data to obtain image modal object category data. Finally, based on the video frame data, a text description is performed on the background image to obtain background description text, and named entity recognition is performed on the background description text to obtain text modal object category data, thereby reducing the degree of manual participation in video background object recognition and realizing multimodal recognition of the categories of objects contained in the background image. Then, based on the video modal object category data, the image modal object category data and the text modal object category data, background object category recognition is performed on the current video stream data, and the multimodal object type recognition results are integrated to improve the recognition accuracy of objects in the video background.

[0086] In step S101 of some embodiments, the background image refers to the remaining image portion after the human body image is cut from the video data or image data. The current video stream data may be the data generated when the user is performing video. For example, in the financial field, the video data generated when the user communicates with financial business personnel via video may be the current video stream data.

[0087] The current video stream data is data generated in real time during video viewing. Therefore, when a user is viewing a video, the camera can capture the image in the user's lens, thereby obtaining the current video stream data including the background image in real time.

[0088] In step S102 of some embodiments, the DETR (DEtection TRansformer) model can be used to identify different objects appearing in the video. Therefore, the present application can use a pre-trained DETR model to identify objects in the background image in the current video stream data, thereby obtaining video modality object category data.

[0089] For details, see Figure 2In some embodiments, step S102 may include but is not limited to steps S201 to S206:

[0090] Step S201, extracting frames from the current video stream data to obtain an initial video frame sequence;

[0091] Step S202, performing image preprocessing on the initial video frame sequence to obtain a standard video frame sequence;

[0092] Step S203, extracting features from the standard video frame sequence to obtain a feature map sequence;

[0093] Step S204, performing attention weight allocation on the feature map sequence to obtain the feature map attention weight;

[0094] Step S205, performing object detection on the feature map sequence based on the feature map attention weight to obtain the object bounding box;

[0095] Step S206: Generate video modality object category data based on the object bounding box.

[0096] In step S201 of some embodiments, video frames in the current video stream data are extracted according to a preset video frame extraction time interval, and multiple initial video frames can be obtained. For example, if the preset video frame extraction time interval is 2 seconds, and the initial video frame at a time of 1 second has been extracted from the current video stream data, then the time of the next extracted initial video frame is 3 seconds. Furthermore, the multiple initial video frames are sorted in chronological order to obtain an initial video frame sequence.

[0097] In step S202 of some embodiments, by performing image preprocessing such as image scaling on the initial video frames in the extracted initial video frame sequence, a standard video frame sequence can be obtained, so that the standard video frame sequence meets the input conditions of the DETR model.

[0098] In step S203 of some embodiments, the DETR model may be used to perform convolution processing and pooling processing on the standard video frame sequence, thereby extracting features from the standard video frame sequence and obtaining a feature map sequence.

[0099] In step S204 of some embodiments, the DETR model includes an attention mechanism layer in the form of a transformer architecture. Therefore, the present application can use the attention mechanism layer in the DETR model to assign attention weights to objects in each feature map in the feature map sequence to obtain the feature map attention weights.

[0100] In step S205 of some embodiments, according to the above-mentioned assigned feature map attention weights, the DETR model can dynamically adjust the focus of the model to detect objects in the feature map, and lock the bounding box of the detected object to obtain the object bounding box.

[0101] In step S206 of some embodiments, the DETR model is used to perform type recognition on the object within the object boundary box, so that the type of the object in the background image in the video modality, that is, the video modality object category data, can be obtained.

[0102] In steps S201 to S206 shown in the present embodiment, frame extraction is performed on the current video stream data to obtain an initial video frame sequence, and then image preprocessing is performed on the initial video frame sequence to obtain a standard video frame sequence. Next, feature extraction is performed on the standard video frame sequence to obtain a feature map sequence, and attention weights are assigned to the feature map sequence to obtain feature map attention weights. Finally, object detection is performed on the feature map sequence based on the feature map attention weights to obtain object bounding boxes, and then video modality object category data is generated based on the object bounding boxes, thereby reducing the degree of manual participation in video background object recognition and realizing object recognition of background images in the video dimension.

[0103] In step S103 of some embodiments, the method of extracting video frames from the current video stream data to obtain video frame data is similar to the method in step S201 above, so it is not repeated here. It should be noted that the time intervals for extracting video frames of the two may be the same or different.

[0104] In step S104 of some embodiments, the SAM (Segment Anything Model) model can be used to segment the image. Therefore, the present application can perform object recognition on the background image in the video frame data by utilizing the segmentation mask method in the SAM model.

[0105] For details, see Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S304:

[0106] Step S301, extracting features from the video frame data to obtain a video frame feature vector;

[0107] Step S302, performing instance recognition on the video frame feature vector to obtain a background instance;

[0108] Step S303, performing mask processing on the background instance to obtain an instance segmentation mask;

[0109] Step S304: extract information from the instance segmentation mask to obtain image modality object category data.

[0110] In step S301 of some embodiments, the convolution layer and pooling layer of the SAM model can be used to perform convolution processing on the video frame data to obtain high-dimensional video features, and then perform pooling processing on the video features to reduce the dimension of the video features, thereby obtaining a video feature vector.

[0111] In step S302 of some embodiments, the SAM model can distinguish background objects from foreground objects in the video frame data through the attention mechanism, and then identify objects in the background objects to obtain the object category of the background objects, i.e., background instances, where the foreground objects refer to objects other than the background image in the current video stream data.

[0112] In step S303 of some embodiments, the SAM model can output an instance segmentation mask including specific location information and object category information of the background instance by performing a mask operation on the background instance.

[0113] In step S304 of some embodiments, information is extracted from the instance segmentation mask output by the SAM model to obtain the object category of each background instance, that is, the image modality object category data. It should be noted that the image modality object category data refers to the object category of each object in the background image identified under the image modality.

[0114] In steps S301 to S304 shown in the present embodiment, feature extraction is performed on the video frame data to obtain a video frame feature vector, and then instance recognition is performed on the video frame feature vector to obtain a background instance. Furthermore, mask processing is performed on the background instance to obtain an instance segmentation mask, and information extraction is performed on the instance segmentation mask to obtain image modality object category data, thereby reducing the degree of manual participation in video background object recognition and realizing object recognition of the background image in the image dimension.

[0115] In step S105 of some embodiments, a BLIP-2 (Bootstrapping Language-Image Pre-training 2) model may be used to convert images and videos into text descriptions. For example, for a scene containing a sofa and a coffee table, the BLIP-2 model may generate a description: "The customer is in the living room, and there is a sofa and a coffee table."

[0116] The present application can use the above-mentioned BLIP-2 model to extract features from video frame data, and can obtain background features and instance features. Then, based on the feature expressions of the background features and instance features, the BLIP-2 model is used to generate text descriptions of the background features and instance features, i.e., background description text.

[0117] For details, see Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S405:

[0118] Step S401, extracting global features from the video frame data to obtain background features;

[0119] Step S402, extracting local features from the video frame data to obtain instance features;

[0120] Step S403, based on the background features and the instance features, the video frame data is subjected to context semantic understanding to obtain the video frame context information;

[0121] Step S404, performing image region weighting on the video frame data based on the video frame context information to obtain an instance weight;

[0122] Step S405 , generating background description text based on background features, instance features, video frame context information and instance weights.

[0123] In step S401 and step S402 of some embodiments, the BLIP-2 model can obtain background features by segmenting the background area in the video frame data, and then perform feature recognition on objects in the background image to obtain instance features. For example, if in the current video stream data, the user is leaning against a wall and there is a bookshelf near the wall, the background features may be features such as the brightness, saturation, and color of the wall, and the instance features may be detail features such as the edge and texture of the bookshelf.

[0124] In step S403 of some embodiments, the BLIP-2 model can clarify the actual function of the instance feature in the background image based on the common functions of the objects in the instance features and the spatial positions between the instance features. For example, when the instance features in the background image are fitness equipment such as dumbbells and treadmills, based on the common functions of dumbbells and treadmills, it can be known that dumbbells and treadmills are used for exercising. At this time, if the dumbbells are placed on the treadmill, the BLIP-2 model may understand that the dumbbells and treadmills are not used for fitness, but are just stored in the space shown in the background image. If the dumbbells and treadmill are placed separately, the BLIP-2 model It may be understood that dumbbells and treadmills are used for fitness. Furthermore, according to the actual function of the instance features in the background image, the BLIP-2 model can parse the specific purpose of the space shown in the background image, thereby realizing the contextual semantic understanding of the video frame data and obtaining the video frame context information. For example, if the actual function of the dumbbells and treadmills is to be displayed, then the specific purpose of the space shown in the background image may be to place fitness equipment, that is, the space may be a fitness equipment room; if the actual function of the dumbbells and treadmills is to be fit, then the specific purpose of the space shown in the background image may be a space to help users exercise, that is, the space may be a gym.

[0125] In step S404 of some embodiments, based on the above video frame context information, the BLIP-2 model may use a self-attention mechanism to assign different weights to different regions in the background image, thereby obtaining instance weights.

[0126] In step S405 of some embodiments, based on the instance weight, the target area in the background image that needs to be described in detail is identified, and then the background description text of the target area is generated based on the background features, instance features, and video frame context information between the background features and the instance features in the target area. For example, if fitness equipment such as dumbbells, sandbags, and treadmills are placed in the target area, and the background feature is a bright space, the generated background description text may be that the dumbbells, sandbags, treadmills, and other fitness equipment are placed in a bright space for fitness.

[0127] In steps S401 to S405 shown in the present embodiment, global feature extraction is performed on the video frame data to obtain background features, and local feature extraction is performed on the video frame data to obtain instance features. Furthermore, based on the background features and instance features, contextual semantic understanding is performed on the video frame data to obtain video frame context information. Secondly, based on the video frame context information, image region weighting is performed on the video frame data to obtain instance weights. Finally, based on the background features, instance features, video frame context information and instance weights, background description text is generated, thereby realizing image-text conversion of the background image, and facilitating object recognition of the background image from the text dimension.

[0128] In step S106 of some embodiments, after obtaining the background description text of the background image, named entity recognition is performed on the background description text according to pre-set rules to obtain a text modal representation of the object type of the object in the background image, that is, text modal object category data.

[0129] For details, see Figure 5 In some embodiments, step S106 may include but is not limited to steps S501 to S502:

[0130] Step S501, identifying named entities in the background description text to obtain a candidate entity sequence;

[0131] Step S502: Based on a preset background object library, entity screening is performed on the candidate entity sequence to obtain text modality object category data.

[0132] In step S501 of some embodiments, background description phrases are obtained by performing word segmentation processing on the background description text, and then context analysis is performed on the background description phrases in combination with the background description text to obtain information such as the part of speech and meaning of the background description phrases in the context of the background description text. Furthermore, based on the information such as the part of speech and meaning of the background description phrases, background description phrases that meet pre-set conditions are screened out from the background description phrases as candidate entity sequences.

[0133] In step S502 of some embodiments, the background object library may be a pre-set database containing common background objects. For example, the background object library may include phrase entities such as furniture (such as sofas, tables, and bookshelves), electronic devices (such as televisions and computers), and daily necessities (such as coffee cups and plants).

[0134] The present application performs entity matching on the candidate entity sequence and a pre-set background object library, thereby screening out a target entity from the candidate entity sequence and using the object category of the target entity as text modality object category data.

[0135] In steps S501 to S502 shown in the present embodiment, a candidate entity sequence is obtained by identifying named entities in the background description text, and then the candidate entity sequence is screened for entities based on a preset background object library to obtain text modal object category data, thereby reducing the degree of manual participation in video background object recognition and realizing object recognition of background images in the text dimension.

[0136] In step S107 of some embodiments, feature extraction is performed on the video modality object category data, image modality object category data, and text modality object category data, respectively, to obtain video modality object feature vectors, image modality object feature vectors, and text modality object feature vectors, and then feature fusion is performed on the feature vectors to obtain background image object categories in the three modalities of video, image, and text.

[0137] For details, see Figure 6 In some embodiments, step S107 may include but is not limited to steps S601 to S604:

[0138] Step S601, performing image feature extraction on video modality object category data to obtain a video modality object feature vector;

[0139] Step S602, performing image feature extraction on the image modality object category data to obtain an image modality object feature vector;

[0140] Step S603, extracting text features from the text modality object category data to obtain a text modality object feature vector;

[0141] Step S604, performing feature fusion on the video modality object feature vector, the image modality object feature vector and the text modality object feature vector to obtain the background image object category.

[0142] In steps S601 to S603 of some embodiments, the video modality object category data and the image modality object category data are both represented in the form of images. Therefore, image feature extraction models such as convolutional neural networks can be used to perform image feature extraction on the video modality object category data and the image modality object category data to obtain video modality object feature vectors and image modality object feature vectors. Furthermore, the text modality object category data is represented in the form of text. Therefore, large language models such as Bert (Bidirectional Encoder Representations from Transformers) models can be used to perform text feature extraction on the text modality object category data to obtain text modality object feature vectors.

[0143] In step S604 of some embodiments, the video modality object feature vector, image modality object feature vector and text modality object feature vector can be weighted and summed according to pre-assigned feature weights to obtain a target feature vector, and then the object category of the background object in the background image can be determined according to the meaning represented by the target feature vector.

[0144] For details, see Figure 7In some embodiments, step S604 may include but is not limited to steps S701 to S703:

[0145] Step S701, weighting the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain feature weight ratio data;

[0146] Step S702, based on the feature weight proportion data, a weighted sum operation is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a target feature vector;

[0147] Step S703: determining the background image object category based on the target feature vector.

[0148] In step S701 of some embodiments, weights may be assigned to the video modality object feature vectors, image modality object feature vectors, and text modality object feature vectors according to pre-set weight assignment data to obtain feature weight proportion data, or weights may be assigned to the video modality object feature vectors, image modality object feature vectors, and text modality object feature vectors using a trained weight assignment model to obtain feature weight proportion data.

[0149] In step S702 of some embodiments, based on the above-assigned feature weights, i.e., feature weight ratio data, a weighted sum of feature vectors is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a target feature vector.

[0150] In step S703 of some embodiments, vector analysis is performed on the target feature vector to clarify the object category represented by the target feature vector, thereby obtaining the category of the background object in the background image, that is, the background image object category.

[0151] In steps S701 to S703 shown in the present embodiment, feature weight proportion data is obtained by weighting the video modality object feature vector, the image modality object feature vector and the text modality object feature vector. Secondly, based on the feature weight proportion data, a weighted sum operation is performed on the video modality object feature vector, the image modality object feature vector and the text modality object feature vector to obtain a target feature vector. Finally, based on the target feature vector, the background image object category is determined, thereby realizing video background object recognition in the three modes of video, image and text, and improving the accuracy of video background object recognition.

[0152] In steps S601 to S604 shown in the present embodiment, image feature extraction is performed on video modal object category data to obtain a video modal object feature vector, image feature extraction is performed on image modal object category data to obtain an image modal object feature vector, and text feature extraction is performed on text modal object category data to obtain a text modal object feature vector. Furthermore, feature fusion is performed on the video modal object feature vector, the image modal object feature vector and the text modal object feature vector. This can integrate the video background object recognition results in three dimensions of video, image and text, reduce the risk of video background object recognition errors, and thus improve the accuracy of video background object recognition.

[0153] The present application performs object recognition on the background image in the current video stream data based on the acquired current video stream data to obtain video modal object category data, then performs image segmentation on the frequency frame data of the current video stream data to obtain image modal object category data, and finally, performs text description on the background image based on the video frame data to obtain background description text, and performs named entity recognition on the background description text to obtain text modal object category data, thereby reducing the degree of manual participation in video background object recognition and realizing multimodal recognition of the categories of objects contained in the background image, and then performs background object category recognition on the current video stream data based on the video modal object category data, the image modal object category data and the text modal object category data, and integrates the multimodal object type recognition results to improve the recognition accuracy of objects in the video background.

[0154] See also Figure 8 The embodiment of the present application further provides a video background object recognition device, which can implement the above-mentioned video background object recognition method, and the device includes:

[0155] The video data acquisition module 801 is used to acquire the current video stream data including the background image;

[0156] A first object acquisition module 802 is used to perform object recognition on the background image based on the current video stream data to obtain video modality object category data;

[0157] The video frame extraction module 803 is used to extract video frames from the current video stream data to obtain video frame data;

[0158] The second object acquisition module 804 performs image segmentation on the video frame data to obtain image modality object category data;

[0159] Background object description module 805, used to perform text description on the background image based on the video frame data to obtain background description text;

[0160] The third object acquisition module 806 is used to perform named entity recognition on the background description text to obtain text modality object category data;

[0161] The object feature fusion module 807 is used to perform background object recognition on the current video stream data based on the video modality object category data, the image modality object category data and the text modality object category data.

[0162] The specific implementation of the video background object recognition device is basically the same as the specific implementation of the above-mentioned video background object recognition method, and will not be repeated here.

[0163] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned video background object recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0164] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0165] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0166] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video background object recognition method of the embodiment of this application;

[0167] Input / output interface 903, used to implement information input and output;

[0168] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0169] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0170] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0171] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video background object recognition method is implemented.

[0172] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The video background object recognition method, video background object recognition device, electronic device and storage medium provided by the embodiments of the present application obtain current video stream data containing a background image, and perform object recognition on the background image based on the current video stream data to obtain video modal object category data. Further, video frame extraction is performed on the above-mentioned current video stream data to obtain video frame data, and image segmentation is performed on the video frame data to obtain image modal object category data. Secondly, a text description is performed on the background image based on the video frame data to obtain background description text, and named entity recognition is performed on the background description text to obtain text modal object category data. Finally, background object category recognition is performed on the current video stream data based on the above-mentioned video modal object category data, the above-mentioned image modal object category data and the above-mentioned text modal object category data.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for identifying background objects in a video, characterized in that: The method comprises: Get the current video stream data including the background image; Based on the current video stream data, performing object recognition on the background image to obtain video modality object category data; Performing video frame extraction on the current video stream data to obtain video frame data; Performing image segmentation on the video frame data to obtain image modality object category data; Based on the video frame data, a text description is performed on the background image to obtain a background description text; Performing named entity recognition on the background description text to obtain text modality object category data; Based on the video modality object category data, the image modality object category data and the text modality object category data, background object category recognition is performed on the current video stream data.

2. The method according to claim 1, characterized in that The performing object recognition on the background image based on the current video stream data to obtain video modality object category data includes: Performing frame extraction on the current video stream data to obtain an initial video frame sequence; Performing image preprocessing on the initial video frame sequence to obtain a standard video frame sequence; Extracting features from the standard video frame sequence to obtain a feature graph sequence; Performing attention weight allocation on the feature map sequence to obtain feature map attention weight; Based on the feature map attention weights, performing object detection on the feature map sequence to obtain an object bounding box; Based on the object bounding box, the video modality object category data is generated.

3. The method according to claim 1, characterized in that The performing image segmentation on the video frame data to obtain image modality object category data includes: Extracting features from the video frame data to obtain a video frame feature vector; Performing instance recognition on the video frame feature vector to obtain a background instance; Performing mask processing on the background instance to obtain an instance segmentation mask; Information is extracted from the instance segmentation mask to obtain the image modality object category data.

4. The method according to claim 1, characterized in that: The step of performing a text description on the background image based on the video frame data to obtain a background description text comprises: Performing global feature extraction on the video frame data to obtain background features; Extracting local features from the video frame data to obtain instance features; Based on the background features and the instance features, performing context semantic understanding on the video frame data to obtain video frame context information; Based on the video frame context information, performing image region weighting on the video frame data to obtain an instance weight; The background description text is generated based on the background features, the instance features, the video frame context information and the instance weight.

5. The method according to claim 1, characterized in that The performing named entity recognition on the background description text to obtain text modality object category data includes: Identify named entities in the background description text to obtain a candidate entity sequence; Based on a preset background object library, the candidate entity sequence is screened to obtain the text modality object category data.

6. The method according to any one of claims 1 to 5, characterized in that: The performing background object recognition on the current video stream data based on the video modality object category data, the image modality object category data and the text modality object category data includes: Performing image feature extraction on the video modality object category data to obtain a video modality object feature vector; Performing image feature extraction on the image modality object category data to obtain an image modality object feature vector; Performing text feature extraction on the text modality object category data to obtain a text modality object feature vector; Feature fusion is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a background image object category.

7. The method according to claim 6, characterized in that The step of fusing the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a background image object category includes: Performing weight allocation on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain feature weight proportion data; Based on the feature weight proportion data, a weighted sum operation is performed on the video modality object feature vector, the image modality object feature vector, and the text modality object feature vector to obtain a target feature vector; Based on the target feature vector, the background image object category is determined.

8. A video background object recognition device, characterized in that: The device comprises: A video data acquisition module is used to acquire current video stream data including a background image; A first object acquisition module, configured to perform object recognition on the background image based on the current video stream data to obtain video modality object category data; A video frame extraction module, used for extracting video frames from the current video stream data to obtain video frame data; A second object acquisition module, based on performing image segmentation on the video frame data, obtains image modality object category data; A background object description module, used to perform text description on the background image based on the video frame data to obtain background description text; A third object acquisition module is used to perform named entity recognition on the background description text to obtain text modality object category data; The object feature fusion module is used to perform background object recognition on the current video stream data based on the video modality object category data, the image modality object category data and the text modality object category data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video background object recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video background object recognition method according to any one of claims 1 to 7 is implemented.

Images