Identity recognition method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an identity recognition method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: performing feature extraction on each object in a video to obtain a first object feature of each object; constructing an object graph structure of the video by taking the first object feature as a node and taking an interaction relationship of different objects in the video as an edge; based on the object graph structure, updating the first object feature of each object to obtain a second object feature of each object; and for each object, performing identity recognition on the object based on the second object feature of the object to obtain the object identity of the object. According to the invention, the accuracy of identity recognition can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an identity recognition method, device, electronic device, storage medium, and program product. Background Technology

[0002] Artificial Intelligence (AI) is a multidisciplinary field encompassing a wide range of areas, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large models or foundational models, can be fine-tuned and widely applied to downstream tasks across various areas of AI.

[0003] In related technologies, for identity recognition, the object features are usually extracted directly from the object in the video to obtain the object's identity. Therefore, the accuracy of identity recognition depends on the object features obtained from feature extraction. Inaccurate object features obtained from feature extraction will lead to poor accuracy of identity recognition. Summary of the Invention

[0004] This application provides an identity recognition method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of identity recognition.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides an identity recognition method, including: Feature extraction is performed on each object in the video to obtain the first object feature for each object; Using the first object feature as nodes and the interaction relationships between different objects in the video as edges, construct the object graph structure of the video; Based on the object graph structure, the first object feature of each object is updated to obtain the second object feature of each object; For each object, based on the object's second object feature, the object is identified to obtain the object's object identity.

[0006] This application provides an identity recognition device, including: The feature extraction module is used to extract features from each object in the video to obtain the first object feature for each object; The construction module is used to construct the object graph structure of the video using the first object feature as nodes and the interaction relationship between different objects in the video as edges; The update module is used to update the first object feature of each object based on the object graph structure to obtain the second object feature of each object; The identity recognition module is used to identify the object based on the object's second object feature for each object, thereby obtaining the object's object identity.

[0007] This application provides an electronic device, including: Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the identity recognition method provided in the embodiments of this application.

[0008] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the identity recognition method provided in this application.

[0009] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the identity recognition method described in this application.

[0010] The embodiments of this application have the following beneficial effects: By extracting first object features from each object in the video, an initial foundation of object attributes is provided for identity recognition. An object graph structure is constructed using these first object features as nodes and the interaction relationships between different objects as edges. This systematically links the correlation information of each object in the video, breaking the isolation of single object features and providing comprehensive support for subsequent feature updates. Based on the object graph structure, the first object features of each object are updated to obtain second object features. These second object features not only retain the core attributes of the object itself but also incorporate the interaction and correlation attributes with other objects, overcoming the limitation of the first object features which only contain their own attributes. This makes the resulting second object features more discriminative. Identity recognition based on these second object features relies on more complete and discriminative feature data, reducing misjudgments caused by insufficient feature dimensions or incomplete information. By supplementing the key information required for identity recognition, the impact of problems such as confusion between similar objects and incomplete feature information on the recognition results is effectively reduced, thereby significantly improving the accuracy of identity recognition. Attached Figure Description

[0011] Figure 1This is a schematic diagram of the architecture of the identity recognition system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of an electronic device for identity recognition provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 1 ; Figure 4 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 2 ; Figure 5 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 3 ; Figure 6 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 4 ; Figure 7 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 5 ; Figure 8 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 6 ; Figure 9 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 7 ; Figure 10 This is a schematic diagram illustrating the principle of multimodal feature extraction provided in the embodiments of this application; Figure 11 This is a schematic diagram illustrating the principle of scene awareness provided in the embodiments of this application; Figure 12 This is a schematic diagram illustrating the principles of model training and retrieval provided in the embodiments of this application; Figure 13 This is a schematic diagram of the object graph structure provided in the embodiments of this application. Figure 1 ; Figure 14 This is a schematic diagram of the structure of the update model provided in the embodiments of this application; Figure 15 This is a schematic diagram of the structure of the identity recognition model provided in the embodiments of this application; Figure 16 This is a schematic diagram of the object graph structure provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, some embodiments are referred to, which describe a subset of all possible embodiments. However, it is understood that some embodiments may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms first, second, and third are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that first, second, and third may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] 1) Video: This refers to a multimedia data collection containing continuous visual images (video frame sequence) and accompanying audio and subtitle information. Its core characteristics are continuity in the temporal dimension and relevance in the content dimension. Video consists of multiple consecutive images, and there is a temporal correlation between frames in terms of scene, object position, or state. Object information (such as people, objects, scene elements, etc.) in the picture can be extracted through image recognition technology. Video contains synchronized subtitle data (such as dialogue subtitles and explanatory subtitles), and the subtitle content must be semantically related to the object behavior and scene events in the video picture.

[0018] 2) Objects: These are entities or virtual elements that can be identified from video footage and possess independent attributes and distinguishable features. They are the core analytical units for constructing the object graph structure. They must be able to be separated and labeled from video frames using computer vision techniques (such as object detection algorithms and image segmentation algorithms) and have clear visual boundaries (such as the outline of a person or the shape of an object).

[0019] 3) Graph Structure: A non-linear data structure used to represent entities (nodes) and the relationships (edges) between them. It is widely used in social networks, knowledge graphs, recommender systems, biological networks (such as protein interactions), and is a core tool for describing complex relationships in the real world.

[0020] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology: In related technologies, for identity recognition, the object features are usually extracted directly from the object in the video to obtain the object's identity. Therefore, the accuracy of identity recognition depends on the object features obtained from feature extraction. Inaccurate object features obtained from feature extraction will lead to poor accuracy of identity recognition.

[0021] This application provides an identity recognition method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of identity recognition. The following describes exemplary applications of the identity recognition system provided in this application.

[0022] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the identity recognition system 100 provided in the embodiments of this application. The terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0023] Terminal 400 is used by users to access client 410 and display object identities on graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.

[0024] In some embodiments, server 200 can be a standalone physical server, a server cluster or business system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, in-vehicle terminal, etc., but is not limited to these. The electronic device provided in this application embodiment can be implemented as a terminal or a server. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0025] In some embodiments, server 200 extracts features from each object in the video to obtain a first object feature for each object; constructs an object graph structure for the video using the first object feature as nodes and the interaction relationships between different objects in the video as edges; updates the first object feature of each object based on the object graph structure to obtain a second object feature for each object; for each object, performs identity recognition based on the second object feature to obtain the object identity, and sends the object identity to terminal 400.

[0026] In other embodiments, terminal 400 extracts features from each object in the video to obtain a first object feature for each object; constructs an object graph structure for the video using the first object feature as nodes and the interaction relationships between different objects in the video as edges; updates the first object feature for each object based on the object graph structure to obtain a second object feature for each object; and for each object, performs identity recognition based on the second object feature to obtain the object identity, and sends the object identity to server 200.

[0027] As an example, in film and television production, on-site video of the film shooting process is acquired, and features are extracted from each object in the video to obtain the first object feature for each object. These objects include actors, props, lighting equipment, etc. The first object feature includes the actor's facial contours and body postures, the appearance, shape, and texture of the props, and the light color and angle of the lighting equipment. Using the first object feature as nodes and the interaction relationships between different objects in the video as edges, an object graph structure for the video is constructed. When an actor holds a prop in performance, a hand-holding interaction is formed between the actor and the prop; this interaction serves as an edge connecting the corresponding node of the actor and the corresponding node of the prop. When light from the lighting equipment shines on the actor, a light-illumination interaction is formed between the lighting equipment and the actor; this interaction serves as an edge connecting the corresponding node of the lighting equipment and the corresponding node of the actor. These nodes and edges are used to construct the object graph structure of the film shooting video. Based on the object graph structure, the first object feature of each object is updated to obtain the second object feature for each object. In the object graph structure, by combining the interaction between the actor and the prop in handheld use, the characteristics of the prop's grip position in the actor's hand are added to the actor's first object characteristics. Simultaneously, the characteristics of the force of the actor's hand movements on the prop are added to the prop's first object characteristics. Similarly, by combining the interaction between the lighting equipment and the actor in terms of light illumination, the characteristics of the light and shadow distribution formed on the actor's body are added to the actor's first object characteristics. The characteristics of the actor's body's reflection of light are added to the lighting equipment's first object characteristics, thus obtaining the second object characteristics for each object. For each object, based on its second object characteristics, the object is identified to obtain its object identity. For actors, based on the second object characteristics including grip position and light and shadow distribution, combined with the actor information file of the film and television production team, the actor's specific identity and role are identified. For props, based on the second object characteristics including the force of manipulation, compared with the film and television prop management list, the name, purpose, and scene of the prop are identified. For lighting equipment, based on the second object characteristics including the degree of reflection, combined with the lighting equipment's configuration information, the model, power, and functional positioning in the shooting scene are identified.

[0028] As an example, in an urban intelligent transportation scenario, traffic detection videos of urban road intersections are acquired. Features are extracted from each object in the video to obtain a first object feature for each object. These objects include passing vehicles, pedestrians, traffic lights, etc. The first object feature encompasses the vehicle type, license plate color, and body color; the pedestrian's walking direction and body outline; and the number of traffic light groups and their current illuminated color. Using the first object feature as nodes and the interaction relationships between different objects in the video as edges, an object graph structure for the video is constructed. When a vehicle travels according to the traffic light's instructions, a following interaction relationship is formed between the vehicle and the traffic light, and this interaction relationship serves as an edge connecting the corresponding node of the vehicle and the corresponding node of the traffic light. When a pedestrian walks on the crosswalk and avoids a vehicle, a yielding interaction relationship is formed between the pedestrian and the vehicle, and this interaction relationship serves as an edge connecting the corresponding node of the pedestrian and the corresponding node of the vehicle. The object graph structure of the road intersection traffic detection video is constructed through these nodes and edges. Based on the object graph structure, the first object feature of each object is updated to obtain a second object feature for each object. In the object graph structure, combining the interaction between vehicles and traffic lights, the acceleration or deceleration characteristics of vehicles according to traffic light instructions are added to the first object feature of the vehicle. Simultaneously, the influence of traffic light instructions on vehicle traffic conditions is added to the first object feature of the traffic light. Similarly, combining the interaction between pedestrians and vehicles, the body turning angle characteristics of pedestrians when avoiding vehicles are added to the first object feature of the pedestrian. The response speed characteristics of vehicles to pedestrian avoidance actions are also added to the first object feature of the vehicle, thus obtaining the second object feature of each object. For each object, based on its second object feature, the object is identified to obtain its object identity. For vehicles, based on the second object features, which include acceleration / deceleration characteristics and response speed characteristics, and combined with the vehicle management database, the vehicle's license plate number, the company to which the vehicle belongs, and the vehicle type, such as a private car, taxi, or bus, are identified. For pedestrians, based on the second object features, which include body turning angle characteristics, and combined with pedestrian flow statistics at the intersection, the direction of travel of pedestrians and whether they are commuters on a fixed route are identified. For traffic lights, based on the second object features, which include characteristics of their impact on vehicles, and in accordance with the traffic light control scheme, the current working mode and control cycle of the traffic lights are identified.

[0029] As an example, in a shopping mall security detection scenario, real-time detection video of the mall is acquired, and features are extracted from each object in the video to obtain the first object feature for each object. These objects include pedestrians entering the mall, shelves within the mall, and moving shopping carts. The first object feature encompasses the pedestrian's height, clothing color, and hairstyle; the shelf's dimensions and placement; and the shopping cart's outline shape and color. Using the first object feature as nodes and the interaction relationships between different objects in the video as edges, an object graph structure for the video is constructed. When a pedestrian pushes a shopping cart, a pushing interaction is formed between the pedestrian and the shopping cart, and this interaction serves as an edge connecting the pedestrian's corresponding node and the shopping cart's corresponding node. When a pedestrian stops to select items from a shelf, a stopping and selecting interaction is formed between the pedestrian and the shelf, and this interaction serves as an edge connecting the pedestrian's corresponding node and the shelf's corresponding node. The object graph structure of the mall detection video is constructed using these nodes and edges. Based on this object graph structure, the first object feature for each object is updated to obtain the second object feature for each object. In the object graph structure, by combining the interaction between pedestrians and shopping carts, the trajectory features of the shopping cart moving with the pedestrian can be added to the pedestrian's first object features, and the force direction features of the pedestrian on the shopping cart can also be added to the shopping cart's first object features. By combining the interaction between pedestrians and shelves, the features of the parts of the goods touched by the pedestrian on the shelves can be added to the shelf's first object features, thus obtaining the second object features of each object. For each object, based on the object's second object features, the object is identified to obtain the object's identity. For pedestrians, based on the second object features containing movement trajectory features, combined with the mall's preset pedestrian identity information database, the pedestrian's identity is identified, such as mall staff, members, or temporary visitors. For shelves, based on the second object features containing features of the parts of the goods touched, compared with the mall's shelf management information, the specific shelf number and the corresponding product category are identified. For shopping carts, based on the second object features containing features related to the force direction, combined with the shopping cart's registration information, the area and usage status of the shopping cart are identified.

[0030] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for identity recognition provided in an embodiment of this application, wherein, Figure 2 The electronic device 500 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 2The illustrated electronic device 500 includes at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0031] Processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0032] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 430.

[0033] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0034] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0035] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, WiFi, and Universal Serial Bus.

[0036] In some embodiments, the identity recognition device provided in this application can be implemented in software. Figure 2 An identity recognition device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: feature extraction module 4551, construction module 4552, update module 4553, and identity recognition module 4554. These modules are logically linked and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0037] In other embodiments, the identity recognition device provided in this application can be implemented in hardware. As an example, the identity recognition device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the identity recognition method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0038] In some embodiments, the terminal or server can implement the identity recognition method provided in this application by running a computer program or computer-executable instructions. For example, the computer program can be a native program in the operating system (e.g., a dedicated identity recognition program) or a software module, such as an identity recognition module that can be embedded in any program (e.g., an instant messaging client, a photo album program, an electronic map client, a navigation client); or it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer program can be any form of application, module, or plugin.

[0039] The identity recognition method provided in this application will be described in conjunction with exemplary applications and implementations of the server or terminal provided in the embodiments of this application.

[0040] See Figure 3 , Figure 3 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 1 , will combine Figure 3Steps 101 to 104 are described below. The identity recognition method provided in this application embodiment can be implemented by the server or the terminal alone, or by the server and the terminal working together. The following description will take the implementation by the server alone as an example.

[0041] In step 101, feature extraction is performed on each object in the video to obtain the first object feature for each object.

[0042] In some embodiments, the first object feature refers to the set of feature data obtained from the inherent, initially presented attribute information of each object in the video through feature extraction methods when processing video content. Feature extraction is based on the objective existence state of each object in the video, focusing on attributes that can be directly observed or obtained through basic analysis when the object is presented independently in the video. The specific content covered by the first object feature varies depending on the type of object in the video application scenario. For example, in a security detection scenario, attributes related to the pedestrian's appearance, such as height, clothing color, and hairstyle, can be extracted for pedestrians; attributes related to the shelf's shape and spatial distribution, such as size and placement, can be extracted for shelves; and attributes related to the shopping cart's appearance, such as outline shape and color, can be extracted for shopping carts. In a film and television production scenario, attributes related to the actor's image and actions, such as facial contours and body posture, can be extracted for actors; attributes related to the prop's shape and material texture, such as appearance shape and material texture, can be extracted for props; and attributes related to the lighting equipment's luminous characteristics and working status, such as light emission color and illumination angle, can be extracted for lighting equipment. In a smart transportation scenario, attributes related to the vehicle's appearance and markings, such as vehicle type, license plate color, and body color, can be extracted for vehicles; attributes related to the pedestrian's movement and appearance, such as walking direction and body outline, can be extracted for pedestrians; and attributes related to the traffic light's structure and working status, such as the number of light groups and the current illuminated color, can be extracted for traffic lights.

[0043] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 2 , Figure 3 Step 101 shown can be achieved through Figure 2 Steps 1011 to 1014 shown are implemented.

[0044] In step 1011, feature extraction is performed on the audio in the video to obtain the first audio feature of the video, and feature extraction is performed on the subtitles in the video to obtain the first text feature of the video.

[0045] In some embodiments, the first audio feature refers to the set of feature data obtained from the inherent, initially presented attribute information of the audio portion contained in the video through audio feature extraction methods during video content processing. Feature extraction is based on the objective existence of the audio in the video and does not rely on the content information of the subtitles in the video. It focuses only on the attributes of the audio that can be directly observed or obtained through basic audio analysis techniques when it is presented independently during video playback. The specific content covered by the first audio feature varies depending on the type of audio in the video application scenario. For example, in a security detection scenario, for the audio in the video, the frequency range of the sound can be extracted, such as the low-frequency distribution of environmental noise and the mid-frequency range of human conversation; the intensity of the sound can be extracted, such as the high-intensity peak of sudden abnormal noise and the low-intensity average of normal environmental sound; the duration of the sound can be extracted, such as the short duration of brief human communication and the duration of equipment operating noise; and the type features of the sound can also be extracted, such as the features that distinguish different types of audio, such as human voices, vehicle horns, and equipment operating sounds. In film and television production, audio from videos can be used to extract the timbre characteristics of human voices, such as the unique timbre of actors' lines and the differences in timbre among voice actors; it can also extract the melodic features of music, such as the melody direction of background music and the rhythmic changes of incidental music; it can extract the attributes of sound effects, such as the frequency changes of scene special effects sounds and the intensity peaks of action sound effects; and it can also extract the channel information of audio, such as the distribution characteristics of mono, stereo, or multi-channel audio. In intelligent transportation scenarios, audio from videos can be used to extract the sound features of vehicles in motion, such as the frequency of car engine roars and the sound intensity of motorcycle exhaust pipes; it can extract sound features related to traffic control, such as the frequency of traffic police whistles and the sound intensity of traffic broadcasts; and it can also extract sound features of the road environment, such as the frequency of pedestrian footsteps crossing the road and the duration of mechanical noise from road construction. The first audio feature, as the initial feature data of the audio dimension in video processing, can provide basic audio support for subsequent comprehensive analysis combining object features and text features in the video. For example, in the process of identity recognition, audio features can be combined to help determine the identity of the object, or in the process of scene understanding, audio features can be used to supplement scene information. The quality of its acquisition directly affects the accuracy of subsequent multi-dimensional feature fusion analysis.

[0046] In some embodiments, the first text feature refers to the set of feature data obtained from the inherent, initially presented attribute information of the subtitles themselves through text feature extraction methods when processing video content. The extraction of the first text feature is based on the objective existence of the subtitles in the video, without relying on the audio information in the video. It focuses only on the attributes of the subtitles that can be directly identified or obtained through basic text analysis techniques when they are presented independently during video playback. The specific content it covers varies depending on the type of subtitle in the video application scenario. For example, in a security detection scenario, if the video contains timestamp subtitles, location marker subtitles, etc., the text content information in the subtitles can be extracted, such as the specific time displayed by the timestamp and the specific area name of the location marker; the text length features of the subtitles can be extracted, such as the fixed character length of the timestamp subtitles and the variation in the number of characters in the location marker subtitles; the text format features of the subtitles can be extracted, such as the font type, font size, and text color; and the frequency features of the subtitles can also be extracted, such as the number of times a specific location marker subtitle appears per unit time and the frequency of the timestamp subtitle's interval. In film and television production, for subtitles and annotations in videos, semantic features can be extracted, such as the emotional tone expressed by subtitles and the explanatory information contained in annotations; structural features can be extracted, such as sentence length and paragraph division in subtitles; language type features can be extracted, such as the identification features of Chinese, English, or other languages; and display position features can also be extracted, such as the characteristics of different display areas like bottom center or bottom left. In intelligent transportation scenarios, if the video contains traffic sign subtitles and prompt subtitles, key information can be extracted, such as the speed limit displayed in traffic sign subtitles and the warning content contained in prompt subtitles; text clarity features can be extracted, such as the legibility of text under different lighting conditions; and update frequency features can be extracted, such as the real-time update interval of traffic prompt subtitles and the fixed display status of sign subtitles. The first text feature, as the initial feature data of the text dimension in video processing, can provide the basic text-level support for subsequent comprehensive analysis combining object features and audio features in the video. For example, in the process of object identity association, text features can be used to supplement the text information related to the object, or in the process of scene event judgment, text features can be used to interpret the event-related description. The quality of its acquisition directly affects the completeness of subsequent multi-dimensional feature fusion analysis.

[0047] In step 1012, object recognition is performed on the video to obtain multiple objects in the video, and steps 1013 to 1014 are performed for each object.

[0048] In some embodiments, object recognition in a video to obtain multiple objects in the video refers to the technical process of detecting, distinguishing and confirming various entities with independent forms, functions or attributes contained in the continuous playback of a video through image recognition technology and video frame analysis, and finally determining the multiple independent entities existing in the video.

[0049] In some embodiments, the object recognition process is based on consecutive frames of a video. By analyzing the pixel information, color distribution, contour structure, and texture features of each frame, and combining this with the entity motion trajectories and morphological change patterns between adjacent frames, the process determines whether there are entities with independent characteristics in the video. Furthermore, it distinguishes the boundaries and attributes of different entities, thereby confirming the existence of each entity as an independent object. The recognition process does not rely on subsequent feature extraction or interaction analysis; it focuses solely on accurately detecting and confirming the existence of various independent entities from the video images. This ensures that each identified object can be clearly distinguished from other objects, and that the shape, position, or basic attributes of each object can be stably captured within the video frames.

[0050] In some embodiments, the continuous video is first divided into a series of independent static frame images using video frame splitting technology. Then, an object detection algorithm is used to scan each frame image to identify possible entity regions in the image. Edge detection, contour extraction, and other methods are used to determine the boundary range of the entity and eliminate the interference of background regions on entity recognition. Next, feature matching technology is used to compare the detected entity regions with a preset object type database to confirm the basic type of the entity, such as people, vehicles, equipment, etc. Finally, the positional changes and morphological consistency of entities in adjacent frame images are combined to determine the continuity of the same entity in different frames, avoiding the repeated identification of the same entity as multiple objects, while ensuring that no new entities appearing in the video are missed, ultimately obtaining multiple independent objects contained in the video.

[0051] In some embodiments, the object recognition process exhibits corresponding scene adaptability in different application scenarios. For example, in security detection scenarios, when performing object recognition on detection videos of shopping malls, streets, etc., objects such as pedestrians, vehicles, shelves, shopping carts, and traffic lights in the video can be identified. Pedestrians are identified by their distinct human outlines and limb movement characteristics, vehicles are identified by their specific body structure, wheel shape, and movement trajectory, and shelves and shopping carts are identified by their fixed shape and placement. Movement status is distinguished; in film and television production scenarios, when performing object recognition on video of the shooting location, objects such as actors, shooting props, lighting equipment, and cameras in the video can be identified. Actors are identified by facial features and body movement features, shooting props are distinguished by specific appearance and material texture, and lighting equipment is identified by luminous areas and equipment outlines; in intelligent transportation scenarios, when performing object recognition on video of road intersection detection, objects such as cars, buses, trucks, pedestrians, traffic signs, and road guardrails in the video can be identified. Different types of vehicles are distinguished by body size and vehicle structure, pedestrians are identified by human body outline and walking trajectory, and traffic signs and road guardrails are identified by fixed shape features and installation positions.

[0052] In some embodiments, the multiple objects obtained through the object recognition process are the basic prerequisites for subsequent object feature extraction, object graph structure construction, and object identity recognition. The accuracy and completeness of the recognition directly affect the effectiveness of all subsequent video processing steps, ensuring that subsequent technical processes can be carried out on specific objects and avoiding deviations in subsequent processing results due to object recognition omissions or errors.

[0053] In step 1013, a target video frame including the object is determined from the video frames of the video, and the image features of the object are determined based on the target video frame.

[0054] In some embodiments, determining the target video frame containing the object from the video frames refers to a technical process that, based on multiple objects identified through object recognition in the video, sequentially detects and matches all static frame images after the video has been segmented, filters out frame images containing at least one identified object, and defines these frame images as target video frames. The process is based on all static frame images after video segmentation, combined with the basic features of each object obtained during the previous object recognition process, such as the object's contour structure, morphological features, and positional information. By comparing the content of each frame image with the features of the identified object, it is determined whether there is a region in the frame image that matches the features of the identified object, thereby determining whether the frame image belongs to the target video frame containing the object. This process does not rely on subsequent feature updates or identity recognition operations; it focuses solely on accurately filtering frame images related to the identified object from a massive number of video frames, ensuring that the target video frame completely covers all time periods when the object appears in the video and does not include blank frames or background frames where no identified object appears. In practice, the feature data of each object recorded during the previous object recognition process is first retrieved, including the object's contour coordinates, morphological parameters, and color distribution range, to establish an object feature comparison library. Then, each frame of static image that has been split is scanned pixel by pixel, and different regions in the frame image are divided using region detection technology to extract the contour, morphological, and color features of each region. Next, the features of each region are matched with the data in the object feature comparison library. If the overlap between the features of a certain region and the features of any identified object reaches a preset threshold, the frame image is determined to contain the corresponding object. Finally, the matching results of all frame images are summarized, and the frame images that are determined to contain at least one identified object are marked as target video frames. At the same time, the object category contained in each target video frame and the specific location of the object in the frame image are recorded to form a set of target video frames. The system exhibits corresponding scene adaptability in different application scenarios. For example, in security detection scenarios, when determining the target video frame from frames of a shopping mall detection video for identified objects such as pedestrians and shopping carts, frames containing pedestrian outlines or shopping cart shapes will be selected. If a frame only shows a shelf background without pedestrians or shopping carts, it will not be included in the target video frame. In film and television production scenarios, when determining the target video frame from frames of a shooting location video for identified objects such as actors and props, frames containing actors' facial features or prop appearances will be selected. If a frame only shows lighting equipment backgrounds without actors or props, it will not be included in the target video frame. In intelligent transportation scenarios, when determining the target video frame from frames of a road intersection detection video for identified objects such as cars and pedestrians, frames containing car body structures or pedestrian walking outlines will be selected. If a frame only shows road guardrails without cars or pedestrians, it will not be included in the target video frame.

[0055] In some embodiments, determining the image features of the object based on the target video frame in step 1013 above can be achieved as follows: In response to the number of target video frames being one, feature extraction is performed on the sub-video frames of the target video frame that include the object to obtain the image features of the object; in response to the number of target video frames being multiple, feature extraction is performed on the sub-video frames of each target video frame that include the object to obtain the sub-video frame features corresponding to each target video frame; the sub-video frame features corresponding to each target video frame are fused to obtain the image features of the object.

[0056] In some embodiments, determining the image features of an object based on a target video frame involves using corresponding feature extraction and processing methods according to the number of target video frames. This allows for the acquisition of feature data that characterizes the object's appearance, shape, and other attributes from the video frame region containing the object. The specific implementation method is described in detail below.

[0057] In some embodiments, when there is only one target video frame, the region containing the object must first be located within that single target video frame, and this region is defined as a sub-video frame. The localization process combines the object's outline structure, positional information, and other basic features obtained during previous object recognition to filter pixel regions in the target video frame that match the object's features, excluding background regions in the frame image that do not contain the object. This ensures that the sub-video frame completely covers the overall shape of the object and does not contain any irrelevant interfering elements. Subsequently, feature extraction is performed on the sub-video frame. The extraction process focuses on the object's appearance attributes and morphological features within the sub-video frame, such as the object's outline details, color distribution patterns, texture structure characteristics, and size proportions. By quantifying and collecting these attributes, feature data that uniquely characterizes the object's state in the current video frame is formed; this feature data is the object's image feature.

[0058] In some embodiments, when there are multiple target video frames, a sub-video frame extraction operation is first performed for each target video frame. For each target video frame, the sub-video frames containing the object are located based on the object's basic features. The location method is consistent with the sub-video frame location logic under a single target video frame, that is, filtering out pixel regions that match the object's features, eliminating background interference, and ensuring that each sub-video frame can accurately and completely present the object in the corresponding target video frame. Then, feature extraction is performed on each sub-video frame to extract features such as the object's outline details, color distribution, texture structure, and size ratio in each sub-video frame, obtaining the sub-video frame features corresponding to each target video frame. These sub-video frame features respectively characterize the object's state features in different target video frames. Then, the features of all sub-video frames are fused. The fusion process needs to comprehensively consider the effectiveness and correlation of the features of each sub-video frame. For example, if the contour features of objects in different sub-video frames are basically the same, the common information of the contour features can be retained; if the color features of an object in a certain sub-video frame show slight differences due to changes in lighting, it needs to be corrected by combining the color features of other sub-video frames. Finally, by integrating and optimizing the features of each sub-video frame, feature data that can comprehensively and accurately represent the overall attributes of the object is formed. This feature data is the image feature of the object.

[0059] In some embodiments, regardless of whether the number of target video frames is one or more, the determination of object image features must be based on the basic features of the object to ensure accurate positioning of sub-video frames, while ensuring the comprehensiveness and accuracy of feature extraction. This will enable the final image features to provide reliable support for subsequent operations such as object identification and object interaction relationship analysis, and avoid errors in subsequent technical processes caused by factor video frame positioning deviations or incomplete feature extraction.

[0060] As an example, in a smart transportation scenario, if the identified object is a sedan, the specific implementation for determining the sedan's image features based on the target video frame is as follows: When there is only one target video frame, it is a static frame image containing the sedan captured at a certain moment by road detection. First, the region containing the sedan is located within the target video frame as a sub-video frame. The localization process combines the sedan's outline structure (such as streamlined body shape and the distribution of the four wheels) and position information (such as driving in the middle lane of the road) to filter out pixel regions that match the features, excluding background elements such as road markings and other surrounding vehicles. Subsequently, features are extracted from this sub-video frame, including the sedan's outline details (such as the roundness of the front and the outline of the trunk), color distribution patterns (such as white body and black windows), texture structure characteristics (such as the smooth paint texture on the body surface and the transparent texture of the window glass), and size ratio parameters (such as the ratio of body length to width and the ratio of wheel diameter to body height). The feature data formed after these attributes are quantified and collected is the image feature of the sedan. When responding to multiple target video frames, these target video frames are multi-frame images captured by road detection during the car's movement. First, for each target video frame, sub-video frames are located based on the car's outline structure and position information, ensuring that each sub-video frame completely contains the car. Next, features are extracted from each sub-video frame. For example, in the frontal frame during driving, the texture of the front grille and the outline around the license plate area are extracted; in the side driving frame, the waistline texture of the car's side profile and the position features of the door handles are extracted; and in the parked frame, the detailed outline of the trunk lid and the complete texture of the wheels are extracted. These are then fused to integrate the outline details from different frames to reconstruct the car's complete appearance. The multi-frame texture information is combined to present the overall texture of the car's paint, grille, windows, etc., correcting color deviations caused by changes in lighting during driving, and preserving the true colors of the car's white body and black windows. Finally, an image feature that comprehensively represents the car's attributes is formed.

[0061] As an example, in a film production scenario, if the identified object is a prop (such as an ancient weapon, a sword), the specific steps for determining the image features of the prop based on the target video frame are as follows: When there is only one target video frame, it is a static frame image containing the sword captured at a certain moment on set. First, the area containing the sword is located within the target video frame as a sub-video frame. The location process combines the previously identified sword's outline structure (such as a slender blade, the shape of the guard connecting the hilt and blade), and positional information (such as the sword being in the center of the frame when held by an actor), filtering out pixel areas that match these features and excluding interfering elements such as the actor's hand and background scenery. Subsequently, features are extracted from this sub-video frame, including sword outline details (such as the straightness of the blade, the curved outline of the guard), color distribution patterns (such as the silver-gray blade and the dark red hilt), texture structure characteristics (such as the metallic luster texture on the blade surface, the texture of the knotted rope on the hilt), and size ratio parameters (such as the length of the blade and the length ratio of the hilt to the blade). The feature data formed by quantifying and collecting these attributes is the image feature of the sword. When responding to multiple target video frames, these target video frames are multi-frame images containing the sword captured from different angles and at different times during the shooting process. First, for each target video frame, sub-video frames are located based on the sword's outline structure and positional information to ensure that each sub-video frame accurately contains the sword. Next, features of each sub-video frame are extracted. For example, in sub-video frames shot from the front, the complete hilt outline and symmetrical texture of the sword are extracted; in sub-video frames shot from the side, the thickness features of the sword and the three-dimensional texture of the hilt are extracted; and in sub-video frames shot against the light, the bright edge color features of the sword are extracted. These yield the sub-video frame features corresponding to each target video frame. Then, feature fusion is performed. For outline features, the features from the front and side frames are combined to reconstruct the sword's three-dimensional outline; for texture features, texture information from frames at different angles is integrated to present the complete texture of the sword; for color features, color deviations caused by lighting in backlit frames are corrected, preserving the true colors of the silver-gray sword and the dark red hilt, ultimately forming a comprehensive and accurate image feature set of the sword.

[0062] Thus, the method of determining object image features based on target video frames can flexibly adopt corresponding processing strategies according to the number of target video frames. When there is only one target video frame, by accurately locating the sub-video frame containing the object and extracting features, the accurate image features of the object at the current moment can be quickly obtained, avoiding interference from background elements and ensuring the effectiveness and relevance of feature data in a single frame scene. When there are multiple target video frames, by extracting features from the sub-video frames of each target video frame separately and performing fusion processing, the feature information of the object at different times, angles, or environments can be integrated, correcting possible deviations in single-frame features (such as contour distortion caused by shooting angle, changes in lighting, etc.). This approach minimizes color deviations and other artifacts, comprehensively capturing the complete attributes of objects, resulting in more complete and reliable image features. This differentiated processing method can adapt to variations in the number of target video frames in different application scenarios. Whether it's a single target video frame formed by the brief appearance of a shelf in security inspection, multiple target video frames formed by multi-angle shooting of props in film production, or multiple target video frames formed during vehicle movement in intelligent transportation, it can effectively output object image features that meet the needs of the scenario. This provides high-quality data support for subsequent operations such as object identification and object interaction relationship analysis based on image features, reducing subsequent processing errors caused by inaccurate or incomplete features, and improving the efficiency and accuracy of the overall video processing workflow.

[0063] In step 1014, a first object feature of the object is determined based on the image features, the first audio features, and the first text features.

[0064] In some embodiments, the determination of the first object feature of the object based on the image features, the first audio features, and the first text features can be achieved by: determining a second audio feature that matches the image features from the first audio features, and determining a second text feature that matches the image features from the first text features; and fusing the image features, the second audio features, and the second text features to obtain the first object feature of the object.

[0065] In some embodiments, determining the first object feature based on image features, first audio features, and first text features involves filtering out audio and text information directly associated with the object through feature matching, then integrating multi-dimensional features to form a first object feature that comprehensively represents the object's attributes, and finally determining the second audio feature that matches the image features from the first audio features. This process requires establishing association matching rules between image features and the first audio features, and combining the attributes of the object presented in the image features (such as object type, action state, and scene) to filter audio information in the first audio features that is directly related to these attributes. For example, if the object corresponding to the image features is a pedestrian and is in a conversational state, and the first audio features contain information such as environmental noise, vehicle horns, and human conversations, then the audio information of conversations matching the human conversations will be filtered out, while irrelevant environmental noise and vehicle horns will be excluded. This filtered audio information is the second audio feature. If the object corresponding to the image features is a vehicle and is in a driving state, and the first audio features contain information such as engine roar, music, and pedestrian footsteps, then the audio information of engine roar matching the vehicle's driving state will be filtered out as the second audio feature.

[0066] In some embodiments, a second text feature matching the image feature is determined from the first text feature. This process requires establishing a matching logic with the first text feature based on the attributes of the object in the image feature (such as object identifier, scene, associated events, etc.), and filtering text information in the first text feature that is directly related to the object attributes. For example, if the object corresponding to the image feature is a shopping mall shelf and is labeled with the location information of the food section, and the first text feature contains subtitle information such as timestamps, location labels, and product names, the text information of the location labels (such as the first floor food section) and the corresponding product names (such as snacks) associated with the food section shelf will be filtered out, while irrelevant timestamp information will be excluded. This filtered text information is the second text feature. If the object corresponding to the image feature is an ancient sword used as a movie prop and is shown in a scene where an actor is holding it, and the first text feature contains information such as dialogue subtitles, prop annotations, and scene descriptions, the text information of the prop annotations (such as a bronze sword, the form of the Warring States period) and scene descriptions (such as props in a battlefield fighting scene) associated with the ancient sword will be filtered out as the second text feature.

[0067] In some embodiments, image features, second audio features, and second text features are fused to obtain the first object feature of the object. The fusion process requires constructing a multi-feature integration model to ensure effective association and complementarity of features across different dimensions: retaining core information in each feature that can independently characterize object attributes, such as object outlines and color textures in image features, sound type and intensity in second audio features, and object identification and scene description in second text features; establishing correlation mappings between features so that features of different dimensions form a mutually corroborating whole, for example, associating pedestrian conversation actions in image features with human conversation sounds in second audio features and conversation subtitles in second text features to jointly characterize the interactive communication attributes of pedestrians; associating vehicle driving status in image features with engine roaring sounds in second audio features and vehicle model labeling in second text features to jointly characterize the vehicle's operation and identification attributes. Through this integration method, scattered multi-dimensional features are transformed into a unified feature set that can comprehensively and systematically characterize object attributes; this set is the first object feature of the object.

[0068] As an example, in a smart transportation scenario, if the object is a bus, its first object feature is determined based on the bus's image features, the first audio feature of the video, and the first text feature. Specifically, the second audio feature matching the image features is determined from the first audio features. The bus's image features include a blue body, the Route 101 sign, and the bus being stopped to pick up and drop off passengers. The first audio features cover information from the road environment, such as car horns, pedestrian footsteps, bus announcements, engine sounds, and traffic light signals. Based on the bus's stopping status in the image features, a matching rule is established to filter sound information related to bus operation from the first audio features, excluding irrelevant audio such as car horns and pedestrian footsteps. Finally, the announcement of the bus arriving at People's Square Station, asking passengers to prepare to disembark, is determined as the second audio feature. Simultaneously, the second text feature matching the image features is determined from the first text features. The first text feature includes information such as roadside electronic screen text (e.g., "Route 101 terminus, Railway Station, traffic congestion ahead, please detour"), bus body electronic screen text (e.g., "Route 101, People's Square → Railway Station"), and timestamp text (e.g., 07:30:15, 07:30:16). Combining the attributes of Route 101 bus in the image features, a matching logic is established to filter information related to Route 101 in the first text feature, excluding irrelevant text such as timestamps and traffic congestion warnings. Finally, the electronic screen text for Route 101 (People's Square → Railway Station) is determined as the second text feature. Subsequently, the image features, the second audio features, and the second text features are fused. The system retains visual information from the image features, such as the bus's blue body, the Route 101 sign, and the location of the bus stop. It also retains the second audio feature, which includes the announcement of the bus arriving at People's Square and the clarity of the audio. Finally, it retains the second text feature, which includes the route information of Route 101 from People's Square to the Railway Station. The system establishes a mapping relationship between the features, linking the bus stop location with the announcement information at People's Square Station and linking the Route 101 sign with the text annotation of Route 101 from People's Square to the Railway Station. This forms a feature set that can comprehensively represent the attributes of the bus, and this set is the first object feature of the bus.

[0069] Thus, the implementation of determining the first object feature based on image features, first audio features, and first text features, by filtering information matching the image features from the first audio features and first text features to obtain the second audio features and second text features, can effectively eliminate audio and text interference data unrelated to the object, ensuring that the included audio and text information is directly related to the object attributes, avoiding feature redundancy or bias caused by irrelevant information; then, the image features are fused with the filtered second audio features and second text features, which not only retains the object's visual attributes (such as appearance, action, and location in image features) but also supplements the object's auditory attributes (such as sound content and type in the second audio features) and text attributes (such as identifiers and scene annotations in the second text features), making the first object feature encompass multi-dimensional information. This multi-feature fusion approach provides a more comprehensive and accurate representation of object attributes. For example, in security detection scenarios, it can combine pedestrian visual features, dialogue audio, and regional text annotations to fully present the pedestrian's status. In film and television production scenarios, it can accurately define prop attributes through armor visual features, collision audio, and prop text annotations. In intelligent transportation scenarios, it can comprehensively reflect bus information based on bus visual features, station announcement audio, and route text annotations. This multi-feature fusion approach can adapt to the feature requirements of objects in different application scenarios, providing richer and more reliable basic data for subsequent construction of object graph structures, updating object features, and conducting identity recognition. It reduces the information loss or misjudgment problems that may exist in single-dimensional features, improves the accuracy and effectiveness of object-related operations in the overall video processing workflow, and reduces invalid data processing and computational resource consumption through feature matching and filtering.

[0070] In some embodiments, the first audio feature includes first sub-audio features of multiple playback moments of the video, and the image feature includes sub-video frame features corresponding to the target video frame.

[0071] In some embodiments, determining the second audio feature matching the image feature from the first audio features can be achieved as follows: for each first sub-audio feature in the first audio features, comparing the playback time corresponding to the first sub-audio feature with the playback time corresponding to the sub-video frame feature to obtain a comparison result corresponding to the first sub-audio feature; in response to the comparison result indicating that the playback time corresponding to the first sub-audio feature is the same as the playback time corresponding to the sub-video frame feature, determining the first sub-audio feature corresponding to the comparison result as the second sub-audio feature; and determining the second audio feature matching the image feature based on the second sub-audio feature.

[0072] In some embodiments, a playback time comparison operation is performed for each first sub-audio feature in the first audio feature. The first audio feature consists of first sub-audio features at multiple playback times of the video, and each first sub-audio feature corresponds to a unique playback time (e.g., audio segment features corresponding to times such as 1 minute 20 seconds, 1 minute 21 seconds, etc.). The image features include sub-video frame features that correspond to the playback time of the target video frame (e.g., features extracted from frame images at times such as 1 minute 20 seconds, 1 minute 22 seconds, etc.). During the comparison process, the playback time information (e.g., timestamp data) of each first sub-audio feature and the playback time information of each sub-video frame feature need to be retrieved, and the time values ​​of the two are compared one by one to determine whether the corresponding first sub-audio feature and sub-video frame feature exist simultaneously at the same time, forming a comparison result for each first sub-audio feature. This result needs to explicitly indicate whether the two playback times are the same. Then, the second sub-audio feature is selected based on the comparison result. When the comparison result indicates that the playback time corresponding to a certain first sub-audio feature is the same as the playback time corresponding to a certain sub-video frame feature, it means that the first sub-audio feature is audio information generated at the same moment when the object presents the state corresponding to the sub-video frame feature, and has a direct correlation with the current state of the object; in this case, the first sub-audio feature corresponding to the comparison result is determined as the second sub-audio feature. If the comparison result indicates that the playback time of the first sub-audio feature is different from that of all sub-video frame features, it means that the first sub-audio feature is audio generated at a moment when the object does not present the corresponding sub-video frame state (such as ambient noise when the object does not appear in the video frame), and has no direct correlation with image features, and is not included in the scope of the second sub-audio feature. Finally, the second audio feature is determined based on the second sub-audio feature. If the selected second sub-audio feature is a single feature, it is directly used as the second audio feature to match the image feature, ensuring that it can represent the synchronous audio information of the object at the corresponding playback time. If the selected second sub-audio feature is a multiple feature (e.g., there are multiple first sub-audio features with the same playback time and different sub-video frame features), these second sub-audio features need to be integrated. The core information related to the object's state in each second sub-audio feature (e.g., synchronous audio details corresponding to the object's actions at different times) is retained, and duplicate or redundant content between features is eliminated to form a set that can completely cover the synchronous audio information of the object at each corresponding playback time. This set is the second audio feature that matches the image feature.

[0073] As an example, in a film and television production scenario, the object is a camera. The first audio feature includes first sub-audio features at multiple playback moments of the video (such as the camera motor sound at 5 minutes 10 seconds, the actor's dialogue at 5 minutes 11 seconds, the lighting equipment cooling fan sound at 5 minutes 12 seconds, and the camera adjustment button sound at 5 minutes 13 seconds). The image features include sub-video frame features corresponding to the playback moments of the target video frames (5 minutes 10 seconds and 5 minutes 13 seconds, both features extracted from frames where the camera appears in the frame). The second audio feature matching the image feature is determined from the first audio feature. Specifically, a playback moment comparison operation is performed for each first sub-audio feature. The timing information of the camera motor sound at 5 minutes and 10 seconds is retrieved and compared with the timings at 5 minutes and 10 seconds and 5 minutes and 13 seconds of the sub-video frame features. A comparison result showing the same timing as the 5 minutes and 10 seconds sub-video frame feature timing is obtained. The timing information of the actor's dialogue sound at 5 minutes and 11 seconds is retrieved and compared with the two sub-video frame feature timings. A comparison result showing the timing is different. The timing information of the lighting equipment cooling fan sound at 5 minutes and 12 seconds is retrieved and compared with the two sub-video frame feature timings. A comparison result showing the timing is different. The timing information of the camera adjustment button sound at 5 minutes and 13 seconds is retrieved and compared with the sub-video frame feature timing. A comparison result showing the same timing as the 5 minutes and 13 seconds sub-video frame feature timing is obtained. Second sub-audio features are selected based on the comparison results. The 5 minutes and 10 seconds camera motor sound, which matches the corresponding sub-video frame feature timing, is identified as the second sub-audio feature. The 5 minutes and 13 seconds camera adjustment button sound, which also matches the corresponding sub-video frame feature timing, is identified as the second sub-audio feature. The other two first sub-audio features with different timings are not included. The second audio feature is determined based on the second sub-audio feature. The two second sub-audio features are integrated, retaining the continuous frequency and volume features of the camera's motor sound and the short frequency and volume features of the camera's adjustment button sound, forming a set of synchronous audio information containing different operating states of the camera. This set is the second audio feature that matches the camera's image features.

[0074] As an example, in a smart transportation scenario, the object is a traffic light. The first audio feature includes first sub-audio features from multiple playback moments of the video (such as the sound of the traffic light's internal relay switching at 8 minutes and 20 seconds, the vehicle horn at 8 minutes and 21 seconds, the pedestrian footsteps at 8 minutes and 22 seconds, and the traffic light malfunction warning beep at 8 minutes and 23 seconds). The image features include sub-video frame features corresponding to the playback moments of the target video frames (8 minutes and 20 seconds and 8 minutes and 23 seconds, both features extracted from the frames in which the traffic light appears on screen). The second audio feature matching the image features is determined from the first audio features, specifically by performing a playback moment comparison operation for each first sub-audio feature. The timing information of the traffic light internal relay switching sound at 8 minutes and 20 seconds was retrieved and compared with the timings of the sub-video frame features at 8 minutes and 20 seconds and 8 minutes and 23 seconds. A comparison result showing the same timing as the 8 minutes and 20 seconds sub-video frame feature was obtained. The timing information of the vehicle horn sound at 8 minutes and 21 seconds was retrieved and compared with the two sub-video frame feature timings, resulting in a comparison result showing different timings. The timing information of the pedestrian footstep sound at 8 minutes and 22 seconds was retrieved and compared with the two sub-video frame feature timings, resulting in a comparison result showing different timings. The timing information of the traffic light malfunction warning beep at 8 minutes and 23 seconds was retrieved and compared with the sub-video frame feature timings, resulting in a comparison result showing the same timing as the 8 minutes and 23 seconds sub-video frame feature. Based on the comparison results, the second sub-audio feature was selected. The 8 minutes and 20 seconds relay switching sound, matching the corresponding sub-video frame feature timing, was identified as the second sub-audio feature. The 8 minutes and 23 seconds malfunction warning beep, matching the corresponding sub-video frame feature timing, was also identified as the second sub-audio feature. The other two first sub-audio features with different timings were not included. The second audio feature is determined based on the second sub-audio feature. The two second sub-audio features are integrated, retaining the frequency and duration features of the relay switching sound and the frequency and interval features of the fault indication beep sound, forming a set of synchronous audio information containing traffic light under different working states. This set is the second audio feature that matches the traffic light image features.

[0075] Thus, the method of determining the second audio feature matching the image feature from the first audio feature is achieved by comparing the playback time corresponding to each first sub-audio feature in the first audio feature with the playback time corresponding to the sub-video frame feature in the image feature. Only the first sub-audio features with the same playback time are determined as the second sub-audio features, and the second audio feature is determined based on the second sub-audio features. This method can strictly control the correlation between audio features and image features in the time dimension, ensuring that the selected second audio feature is audio information generated at the same time as the object's presentation state in the target video frame. It effectively eliminates the influence of irrelevant audio at different times (such as ambient sound when the object does not appear in the video frame, or interference sound from other time periods) on the matching result, and avoids the influence of time misalignment. This addresses the issue of mismatch between audio and image features, improving the accuracy of the association between the second audio feature and the image feature. This playback-time-based matching method eliminates the need for complex feature attribute comparison algorithms, achieving effective filtering solely through consistency judgment in the time dimension. The operation logic is simple and efficient, reducing the computational load in the feature matching process and lowering the performance requirements of processing equipment. At the same time, it ensures the reliability of the filtering results, enabling the first object feature obtained by fusing image features, second audio features, and second text features to have higher consistency and accuracy in the time dimension. This provides multi-dimensional data support for subsequent construction of object graph structures, updating object features, and conducting identity recognition, further reducing subsequent processing errors caused by feature time asynchrony.

[0076] In some embodiments, the first text feature includes first sub-text features of multiple playback moments of the video, and the image feature includes sub-video frame features corresponding to the target video frame.

[0077] In some embodiments, determining the second text feature matching the image feature from the first text feature can be achieved as follows: for each first sub-text feature in the first text feature, comparing the playback time corresponding to the first sub-text feature with the playback time corresponding to the sub-video frame feature to obtain a comparison result corresponding to the first sub-text feature; in response to the comparison result indicating that the playback time corresponding to the first sub-text feature is the same as the playback time corresponding to the sub-video frame feature, determining the first sub-text feature corresponding to the comparison result as the second sub-text feature; and determining the second text feature matching the image feature based on the second sub-text feature.

[0078] In some embodiments, a playback time comparison operation is performed for each first sub-text feature in the first text features. The first text features consist of first sub-text features at multiple playback times of the video, and each first sub-text feature corresponds to a unique playback time (such as the time corresponding to the area annotation subtitle at 2 minutes and 15 seconds, the time stamp subtitle at 2 minutes and 16 seconds, etc.). The image features include sub-video frame features that correspond to the playback time of the target video frame (such as features extracted from frame images at 2 minutes and 15 seconds, 2 minutes and 18 seconds, etc.). During the comparison process, the playback time information of each first sub-text feature (such as the timestamp data of the subtitle display) and the playback time information of each sub-video frame feature need to be retrieved, and the time values ​​of the two are compared one by one to determine whether the corresponding first sub-text feature and sub-video frame feature exist at the same time, forming a comparison result for each first sub-text feature. This result needs to explicitly indicate whether the two playback times are the same. Then, the second sub-text features are selected based on the comparison results. When the comparison result indicates that the playback time corresponding to a certain first sub-text feature is the same as the playback time corresponding to a certain sub-video frame feature, it means that the first sub-text feature is the text information displayed at the same moment when the object presents the state corresponding to the sub-video frame feature, and has a direct correlation with the current state of the object; in this case, the first sub-text feature corresponding to the comparison result is determined as the second sub-text feature. If the comparison result indicates that the playback time of the first sub-text feature is different from that of all sub-video frame features, it means that the first sub-text feature is the text displayed when the object does not present the corresponding sub-video frame state (such as subtitles marked in irrelevant areas when the object does not appear in a video frame), and has no direct correlation with image features, and is not included in the scope of the second sub-text feature. The second text feature is determined based on the second sub-text feature. If the selected second sub-text feature is a single feature, it is directly used as the second text feature to match the image feature, ensuring that it can represent the synchronous text information of the object at the corresponding playback time. If the selected second sub-text feature is multiple (e.g., there are multiple first sub-text features with the same playback time and different sub-video frame features), these second sub-text features need to be integrated. The core information related to the object's state in each second sub-text feature (e.g., synchronous text annotation details of the scene where the object is located at different times) should be retained, and duplicate and redundant content between features (e.g., repeated subtitle display of the same information) should be eliminated to form a set that can completely cover the synchronous text information of the object at each corresponding playback time. This set is the second text feature that matches the image feature.

[0079] As an example, in a film and television production scenario, the object is a wooden prop table in a period drama scene. The first text feature includes first sub-text features at multiple playback moments of the video (such as the subtitle "Props Table in Period Drama Study - 03" at 8 minutes and 15 seconds, the 8:16:00 timestamp subtitle at 8 minutes and 16 seconds, the subtitle "Props Chair in Period Drama Courtyard - 01" at 8 minutes and 17 seconds, and the supplementary subtitle "Props Table in Period Drama Study - 03 Material: Pine Wood" at 8 minutes and 18 seconds). The image features include sub-video frame features corresponding to the playback moments of the target video frames (8 minutes and 15 seconds and 8 minutes and 18 seconds, both features extracted from the frame images where the wooden prop table appears in the frame). The second text feature matching the image feature is determined from the first text feature, specifically implemented as follows: a playback moment comparison operation is performed for each first sub-text feature. The timing information of the subtitle "03" in Scene 05 (Ancient Costume Study Room Props Table - 03) at 8 minutes and 15 seconds is retrieved and compared with the timing information of the sub-video frame features at 8 minutes and 15 seconds and 8 minutes and 18 seconds. The result shows the same timing as the 8 minutes and 15 seconds sub-video frame feature timing. The timing information of the subtitle "8:16:00" at 8 minutes and 16 seconds is retrieved and compared with the timing information of the two sub-video frame feature timings. The result shows different timing. The timing information of the subtitle "01" in Scene 06 (Ancient Costume Courtyard Props Chair - 01) at 8 minutes and 17 seconds is retrieved and compared with the timing information of the two sub-video frame feature timings. The result shows different timing. The timing information of the subtitle "03" in Scene 05 (Ancient Costume Study Room Props Table - 03, Material: Pine Wood) at 8 minutes and 18 seconds is retrieved and compared with the timing information of the sub-video frame feature timing. The result shows the same timing as the 8 minutes and 18 seconds sub-video frame feature timing. The second sub-text feature is then selected based on the comparison results. The subtitle at 8 minutes 15 seconds in Scene 05, the prop table in the ancient study, and the corresponding sub-video frame feature time is the same as the subtitle at 8 minutes 15 seconds, and this is identified as the second sub-text feature. Similarly, the subtitle at 8 minutes 18 seconds in Scene 05, the prop table in the ancient study, and the material "pine wood" is the same as the sub-video frame feature time, and this is also identified as the second sub-text feature. The first sub-text features, which differ at the other two times, are not included. The second text feature is determined based on the second sub-text feature. The two second sub-text features are integrated, retaining the scene information of Scene 05, the ancient study, and the prop identification information of Prop Table - 03, while supplementing the attribute information of "pine wood," forming a complete information set including the prop table scene, identification, and material. This set is the second text feature matching the image features of the wooden prop table.

[0080] As an example, in a smart transportation scenario, the object is a speed limit traffic sign on a certain road. The first text feature includes first sub-text features of multiple playback moments in the video (such as the subtitle of the 60km / h speed limit traffic sign on the East Ring Road at 12 minutes and 20 seconds, the 12:21:00 timestamp subtitle at 12 minutes and 21 seconds, the 40km / h speed limit traffic sign on the South Ring Road at 12 minutes and 22 seconds, and the supplementary explanation subtitle of the 60km / h speed limit on the East Ring Road effective from 7:00 AM to 8:00 PM at 12 minutes and 23 seconds). The image features include sub-video frame features corresponding to the playback moments of the target video frames (12 minutes and 20 seconds and 12 minutes and 23 seconds are features extracted from the frame images in which the East Ring Road speed limit traffic sign appears in the video). The second text feature that matches the image feature is determined from the first text feature. Specifically, a playback moment comparison operation is performed for each first sub-text feature. The time information of the 60km / h speed limit subtitle on the East Ring Road at 12 minutes and 20 seconds was retrieved and compared with the time information of 12 minutes and 20 seconds and 12 minutes and 23 seconds in the sub-video frame features. The comparison result showed that it was the same as the time information of the 12 minutes and 20 seconds sub-video frame feature. The time information of the 12:21:00 timestamp subtitle at 12 minutes and 21 seconds was retrieved and compared with the time information of the two sub-video frame features. The comparison result showed that the time information was different. The time information of the 40km / h speed limit subtitle on the South Ring Road at 12 minutes and 22 seconds was retrieved and compared with the time information of the two sub-video frame features. The comparison result showed that the time information was different. The time information of the 60km / h speed limit effective from 7:00 AM to 8:00 PM on the East Ring Road at 12 minutes and 23 seconds was retrieved and compared with the time information of the sub-video frame features. The comparison result showed that it was the same as the time information of the 12 minutes and 23 seconds sub-video frame feature. The second sub-text feature was selected based on the comparison results. The subtitle "60km / h speed limit on Donghuan Road" at 12 minutes and 20 seconds, matching the time signature of its corresponding sub-video frame, is identified as the second sub-text feature. Similarly, the subtitle "60km / h speed limit on Donghuan Road effective from 7 AM to 8 PM" at 12 minutes and 23 seconds, also matching the time signature of its corresponding sub-video frame, is identified as the second sub-text feature. The first sub-text features, differing in time signature, are not included. The second text feature is determined based on this second sub-text feature. The two second sub-text features are integrated, retaining the road information of Donghuan Road and the speed limit standard information of 60km / h, while supplementing the effective time period information of 7 AM to 8 PM. This forms a complete information set containing the road where the speed limit sign is located, the speed limit standard, and the effective time period. This set is the second text feature matching the image features of the Donghuan Road speed limit traffic sign.

[0081] Thus, by comparing the playback time of each first sub-text feature with that of the sub-video frame features, only the first sub-text features with the same playback time are identified as second sub-text features and further integrated to form the second text feature. This allows for strict locking of text information that is contemporaneous with the object's presentation state in the target video frame, effectively eliminating interference from irrelevant text at different times (such as annotations in other areas when the object does not appear in the video frame, or simple timestamp subtitles). This avoids the problem of mismatch between text and image features due to time misalignment, significantly improving the accuracy of the association between the second text feature and image feature. This playback time-based matching method does not require complex text semantic and image content association algorithms; effective filtering can be completed solely through time consistency judgment. It is concise and efficient, reducing the computational load in the feature matching process, lowering the performance requirements of processing equipment, and ensuring the reliability of the screening results. By integrating the second sub-text features of time-matching to form the second text features, it can completely cover the synchronous text information of each corresponding playback time of the object, such as the area and product activity information of the snack shelf in the security detection scene, the scene and material information of the prop table in the film and television production scene, and the road and effective time information of the speed limit sign in the intelligent transportation scene. It provides time-synchronized text data support for the subsequent fusion of image features, second audio features and second text features to obtain accurate first object features, further reducing subsequent processing errors caused by feature time asynchrony, and improving the stability and effectiveness of object-related operations in the overall video processing workflow.

[0082] As an example, in a smart urban transportation scenario, the processing of road intersection detection videos follows this process: First, feature extraction is performed on the audio in the video to obtain the first audio feature. This includes vehicle horns at different playback times, bus announcements (e.g., the upcoming Park South Gate stop), pedestrian footsteps, traffic light alerts, and road construction sounds. These audio elements collectively constitute the first audio feature. Simultaneously, feature extraction is performed on the video subtitles to obtain the first text feature. The video subtitles include route labels for different time periods (e.g., bus route 102 and bus route 5), speed limit information (e.g., speed limit 60km / h), and timestamps (e.g., 7:45:00, 7:48:00). These text elements collectively constitute the first text feature. Next, object recognition is performed on the video to identify multiple objects, including bus route 102, private cars, pedestrians, and traffic lights. For bus route 102, the following processing is performed: Target video frames containing the bus are identified from the video frames. Frames containing the bus's blue body and the route 102 sign are selected to obtain the target video frames (e.g., frames from playback times of 7:45:00 and 7:48:00). Based on the target video frames, the bus's image features are determined. Features of the bus are extracted from the sub-video frames, including its streamlined body, blue exterior, the route 102 sign on the front, and its position in the middle lane. These features are then fused to obtain the bus's image features. Based on the image features, first audio features, and first text features, the first object features of the bus are determined. The system selects second audio features from the first audio features that are the same as the timestamps of the image features, such as the bus announcement at 7:45:00 and the bus horn at 7:48:00; it also selects second text features from the first text features that are the same as the timestamps of the image features, such as the route label of bus route 102 at 7:45:00 and the speed limit notice at 60km / h at 7:48:00; and it merges the bus image features with the second audio features and the second text features to form a first object feature that includes the bus's appearance, route, synchronous sound, and speed limit information.

[0083] Thus, the processing flow of extracting audio and subtitle features from videos, recognizing objects, and determining target video frames, image features, and first object features for each object, comprehensively captures sound and text information at different moments in the video by extracting the first audio and first text features of the video separately. This avoids the limitations of single visual dimension information and provides rich multi-dimensional data support for subsequent object feature analysis. By recognizing multiple objects in the video, it is possible to accurately distinguish independent entities in the video content, laying the foundation for personalized processing for each object. Determining the target video frame containing each object eliminates interference from blank frames without objects, focusing on extracting image features of objects from effective frame images, improving the efficiency and accuracy of image feature extraction. Based on image features, first audio features, and first text features, the first object features of the object are determined. By filtering and fusing audio and text information that matches the image features at the same moment, it is ensured that the first object features include both the visual attributes of the object and the auditory and text attributes of the same period, making the feature information more comprehensive and complete.

[0084] In step 102, an object graph structure of the video is constructed using the first object feature as nodes and the interaction relationships between different objects in the video as edges.

[0085] In some embodiments, an object graph structure refers to a structured data model constructed when processing video content. It uses the first object features of each object in the video as the basic building blocks and the actual interactions between different objects as connecting links. This model is used to intuitively present object attributes and relationships between objects. The structure is built based on two core principles: the first object features and object interaction relationships. The first object features, acting as nodes, encompass the object's image features, matching second audio features, and second text features. This comprehensively represents the attribute information of each object. For example, in security detection scenarios, this includes the facial contours of pedestrians, simultaneous dialogue audio, and text annotations of their location; in film and television production scenarios, this includes the appearance, simultaneous sound features, and prop identification text; and in intelligent transportation scenarios, this includes vehicle body features, simultaneous station announcement audio, and route text information. Each object's first object feature corresponds to an independent node in the graph structure, ensuring that each node uniquely identifies the corresponding object.

[0086] In some embodiments, the interaction relationships, acting as edges, originate from the interactions between different objects in the video at the same time or consecutively. Examples include the pushing relationship between pedestrians and shopping carts, and the stopping and selecting relationship between pedestrians and shelves in security detection scenarios; the hand-holding relationship between actors and props, and the lighting relationship between lighting equipment and actors in film production scenarios; and the following instruction relationship between vehicles and traffic lights, and the yielding relationship between pedestrians and vehicles in intelligent transportation scenarios. Each interaction relationship corresponds to an edge between two related object nodes in the graph structure. The attributes of the edges can be defined according to the type of interaction relationship to distinguish different forms of object association. The object graph structure, through the organic combination of nodes and edges, integrates the scattered object attributes and inter-object associations in the video into a unified structured model. This model can intuitively display the complete attributes of a single object and clearly present the interaction network between multiple objects. It provides structured data support for subsequent updates to object features and object identification based on this structure, avoiding the difficulties in association analysis caused by scattered object information. This improves the overall understanding and processing efficiency of objects and their relationships in the video, while also presenting complex video content in a structured form, facilitating the subsequent algorithm's retrieval and analysis of object information.

[0087] As an example, see Figure 13 , Figure 13 This is a schematic diagram of the object graph structure provided in the embodiments of this application. Figure 1 , Figure 13 The object graph structure shown includes node 1, node 2, node 3, node 4 and node 5. There are interactive relationships between node 1 and node 2, node 1 and node 3, node 2 and node 3, node 2 and node 4, and node 3 and node 5.

[0088] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 3 , Figure 3 Before step 102 shown, it is also possible to perform the operation separately for each object in the video. Figure 5 Steps 105 to 106 shown demonstrate the determination of the interactive relationship.

[0089] In step 105, the object is identified as the first object, and the object in the video that is different from the first object is identified as the second object.

[0090] As an example, the objects in the video include object A, object B, and object C; when object A is the first object, the second objects are object B and object C; when object B is the first object, the second objects are object A and object C; when object C is the first object, the second objects are object A and object B.

[0091] In step 106, based on the video frames and subtitles in the video, a third object that has the interaction relationship with the first object is determined from a plurality of second objects.

[0092] In some embodiments, the interaction relationship refers to the state or behavioral connection between a first object and a specific second object, determined during video processing based on the object behavior states presented in the video frames and the association information provided by the video subtitles, and considering the identified first object and other second objects. The determination of the interaction relationship relies on both video frames and video subtitles as core criteria. Video frames can visually present the spatial relationship and behavioral states of the first and second objects. For example, if the first object is a pedestrian, and the video frame shows the pedestrian and another second object (such as a shopping cart) in the same spatial area, with the pedestrian's hand touching the shopping cart handle and moving together, this provides visual evidence for determining an interaction relationship. Video subtitles supplement the association information that video frames cannot directly present. For example, if the first object (such as a shelf) and the second object (such as a promotional sign) are placed adjacent to each other in the video frame, and the subtitles indicate that the shelf corresponds to a promotional sign activity, this can help determine that an interaction relationship exists between the two. The manifestation of interactive relationships varies depending on the type of object in the application scenario. For example, in security detection scenarios, it can manifest as a pushing relationship between pedestrians and shopping carts, a stopping and selecting relationship between pedestrians and shelves, and a corresponding relationship between shelves and promotional signs. In film and television production scenarios, it can manifest as a handheld usage relationship between actors and props, a lighting relationship between lighting equipment and actors, and a shooting correspondence between cameras and scenes. In intelligent transportation scenarios, it can manifest as a following relationship between vehicles and traffic lights, a yielding relationship between pedestrians and vehicles, and a guiding relationship between road signs and vehicles. Determining interactive relationships is a key basis for selecting third objects from multiple second objects. By clarifying the interaction between the first and third objects, a relationship network between objects can be constructed. This provides important relational support for subsequent operations such as object feature updates and deep object identity recognition based on this relationship network, avoiding processing errors caused by the inability to distinguish relationships between objects. At the same time, it makes the behavioral logic of objects in the video clearer and improves the accuracy of overall understanding and analysis of the video content.

[0093] In some embodiments, see Figure 6 , Figure 6This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 4 , Figure 5 Step 106 shown can be performed separately for each second object. Figure 6 Steps 1061 to 1063 shown are implemented.

[0094] In step 1061, based on the first object and the second object, video frames in the video are identified to obtain identification results.

[0095] In some embodiments, the identification of video frames in the video based on the first object and the second object to obtain the identification result can be achieved in the following manner: For each video frame in the video, when the number of objects present in the video frame is one, the video frame is determined as a first video frame; when the number of objects present in the video frame is multiple, and the multiple objects present in the video frame include the first object and the second object, the video frame is determined as a second video frame; in response to all video frames in the video being the first video frame, the identification result is determined to be that no video frame includes the first object and the second object; in response to the presence of the second video frame in the video frame, the identification result is determined to be that a video frame includes both the first object and the second object.

[0096] In some embodiments, a video frame classification operation is performed for each video frame in the video. The classification process first uses object recognition technology to detect the number of objects and the type information of each object in the current video frame, and then classifies the frame according to the detection results: if only one object is detected in the current video frame, regardless of whether the single object is the first object or the second object, the video frame is identified as the first video frame. This type of video frame contains only a single independent object and cannot simultaneously present the coexistence of the first object and the second object; if multiple objects are detected in the current video frame, and it is further confirmed that these multiple objects simultaneously contain both the first object and the second object (i.e., through object type comparison, it is determined that the frame contains both the characteristics of the first object and the characteristics of the second object), then the video frame is identified as the second video frame. This type of video frame can directly present the coexistence of the first object and the second object in the same frame, providing a basis for subsequent determination of their association. The final recognition result is determined based on the classification results of all video frames in the video. The judgment process needs to cover all video frames of the video, checking the existence of each type of video frame one by one: If the check finds that all video frames in the video belong to the first video frame, that is, each video frame either contains only a single object, or contains multiple objects but does not simultaneously contain the first object and the second object, it means that there is no frame image in the entire video that can simultaneously present the first object and the second object. At this time, the recognition result is determined to be that there is no video frame that includes the first object and the second object; If the check finds that there is at least one second video frame in the video, that is, there is at least one frame image that simultaneously contains the first object and the second object, it means that there is a frame image in the video that can present the coexistence of the two objects. At this time, the recognition result is determined to be that there is a video frame that includes the first object and the second object.

[0097] As an example, in a film production scenario, the first object is the actor, and the second object is the ancient sword prop. The video frames of the shooting scene are identified as follows: A classification operation is performed on each video frame. Each video frame is detected using object recognition technology: if a video frame contains only the actor (e.g., an actor reciting lines alone) or only the ancient sword prop (e.g., the sword prop is placed on a prop stand), the video frame is identified as the first video frame; if a video frame contains multiple objects, and these objects include both the actor and the ancient sword prop (e.g., an actor practicing with the sword prop), the video frame is identified as the second video frame. The identification result is determined based on the classification results: if all video frames in the video are found to be first video frames (e.g., the actor and the sword prop never appear in the same frame during the shooting period), the identification result is determined to be that no video frame contains both the actor and the ancient sword prop; if at least one second video frame is found in the video (e.g., a frame image of an actor holding the sword prop), the identification result is determined to be that a video frame contains both the actor and the ancient sword prop.

[0098] As an example, in a smart transportation scenario, the first object is a bus, and the second object is a traffic light. The identification of video frames from road intersection detection videos is implemented as follows: A classification operation is performed on each video frame. Each video frame is detected using object recognition technology: if a video frame contains only one object (e.g., a bus traveling on a road without traffic lights) or only one object (e.g., a traffic light is on but no bus is passing), the video frame is identified as the first video frame. If a video frame contains multiple objects, and these objects include both a bus and a traffic light (e.g., a bus waiting at a traffic light), the video frame is identified as the second video frame. The identification result is determined based on the classification results: if all video frames in the video are first video frames (e.g., a bus and a traffic light never appear in the same frame during the detection period), the identification result is determined to be that no video frame contains both a bus and a traffic light; if at least one second video frame is found in the video (e.g., a frame image of a bus waiting at a traffic light), the identification result is determined to be that a video frame contains both a bus and a traffic light.

[0099] Thus, the method of identifying video frames based on the first and second objects and obtaining results can accurately filter out frames where both types of objects coexist by dividing the video frames into first and second video frames according to the number of objects in the video frame and whether they contain both types of objects simultaneously. This avoids misclassifying single-object frames or multi-object frames containing other objects as frames where both types of objects coexist, significantly improving the accuracy of frame classification. Determining the final identification result based on the presence of a second video frame in all video frames can comprehensively cover the video content, avoiding identification errors caused by missing frames. For example, in security detection scenarios, it can accurately determine whether pedestrians and shopping carts have been in the same frame; in film production scenarios, it can confirm whether actors and shopping carts have been in the same frame. Whether sword props appear simultaneously, and whether buses and traffic lights coexist in intelligent transportation scenarios; the entire process can be completed simply by detecting the number and type of objects, without the need for complex behavior analysis algorithms. The operation logic is simple and efficient, which can reduce the consumption of computing resources and quickly output clear recognition results. This provides a reliable frame-level judgment basis for further analysis of the interaction relationship between the two types of objects (such as determining whether there is a pushing relationship between pedestrians and shopping carts, or whether there is a holding relationship between actors and sword props), reduces invalid operations in subsequent interaction relationship analysis, improves the efficiency and reliability of the overall object association processing flow, and adapts to the needs of object coexistence recognition in various application scenarios.

[0100] In step 1062, in response to the recognition result indicating that the video frame includes the first object and the second object, the subtitles in the video are retrieved to obtain the retrieval result.

[0101] In some embodiments, the triggering conditions and search scope of subtitle retrieval are clearly defined. The triggering condition is that the preliminary identification results have confirmed that there is at least one video frame in the video that simultaneously contains the first object and the second object. In this case, the time range of subtitle retrieval needs to be defined around the playback time corresponding to these video frames in the same frame, ensuring that the retrieval focuses on the time period when the two types of objects may interact and have dialogue, and avoiding invalid retrieval for time periods without frames in the same frame. For example, if the first object (pedestrian) and the second object (shopping cart) are identified in the preliminary identification that they are in the same frame in the video at 10 minutes and 05 seconds and 10 minutes and 08 seconds, then the subtitle retrieval scope is limited to the subtitle content before and after 10 minutes and 05 seconds to reduce interference from irrelevant subtitles. Targeted retrieval of subtitle content is then performed. The retrieval process requires first extracting all subtitle text (including dialogue subtitles, annotation subtitles, etc.) in the video subtitles corresponding to the above time range, and then filtering dialogue content directly related to the first object and the second object through keyword matching or semantic association analysis. Keyword matching requires setting search keywords based on the attribute characteristics of the two types of objects. For example, when the first object is a pedestrian and the second object is a shopping cart, search keywords can include words related to the two types of objects and possible interactive behaviors, such as pedestrian, shopping cart, push, and take. Semantic association analysis requires determining whether there are explicit statements in the subtitle text that relate to the two types of objects. For example, a dialogue in the subtitle asking "Does this shopping cart need to be pushed to the checkout counter?" mentions the second object (shopping cart) and implicitly implies the behavior of the first object (pedestrian), thus belonging to the related dialogue that needs to be screened. The search results determine whether there is dialogue between the two types of objects in the subtitles. If, through searching, at least one dialogue text that explicitly relates the first and second objects is found in the subtitles within a defined time range (such as the dialogue mentioning the shopping cart and pedestrian behavior mentioned above), then the search results are determined to indicate that there is dialogue between the first and second objects in the subtitles. If, after searching, no dialogue text relating to the two types of objects is found, and only subtitles unrelated to the two types of objects exist (such as timestamp subtitles or dialogue subtitles of other objects), then the search results are determined to indicate that there is no dialogue between the first and second objects in the subtitles.

[0102] In step 1063, in response to the search result indicating that there is a dialogue between the first object and the second object in the caption, the second object is identified as a third object that has an interactive relationship with the first object.

[0103] In some embodiments, video frames in the video are identified based on a first object and a second object to obtain an identification result. This step requires detecting the number and type of objects contained in each video frame: if a video frame contains only one object (regardless of whether it is the first object or the second object), it is identified as the first video frame; if a video frame contains multiple objects and simultaneously contains both the first and second objects, it is identified as the second video frame. After checking all video frames, if all frames are first video frames, the identification result is that there are no video frames that simultaneously include both types of objects; if there is at least one second video frame, the identification result is that there are video frames that simultaneously include both types of objects. This step quickly identifies visual scenes where the two types of objects may interact through basic detection of the number and type of objects, providing frame-level basis for subsequent interaction relationship determination. For example, in security detection scenarios, this step can be used to confirm whether a pedestrian and a shopping cart have appeared in the same frame. In response to the identification result indicating the existence of video frames that simultaneously include both types of objects, the subtitles in the video are retrieved to obtain the retrieval results. The search process begins by defining a time range around the playback time of the second video frame to avoid invalid searches for periods without corresponding frames. For example, if the playback time of the second video frame is from 10 minutes 05 seconds to 10 minutes 08 seconds, only the subtitles before and after this time period are searched. Then, all subtitle text within this time range is extracted. Keyword matching (combining the attributes of the two object types to define relevant terms, such as "push" and "shopping cart" in relation to pedestrians and shopping carts) or semantic association analysis (determining whether the subtitles implicitly contain related behaviors between the two objects, such as "Does this shopping cart need to be pushed to the checkout counter?") is used to filter dialogue content directly related to the two objects. If at least one related dialogue is found, the search result indicates that there is dialogue between the two objects in the subtitles; if no related dialogue is found, the search result indicates that such dialogue does not exist. This step, through time range limitation and content association analysis, verifies whether there is linguistic interaction between the two objects in a scene where they are in the same frame, supplementing evidence of association that cannot be directly presented by visual information. In response to the search results indicating the existence of dialogue between the two objects in the subtitles, the second object is identified as a third object that interacts with the first object. When video frame recognition confirms that two types of objects are in the same frame and subtitle retrieval verifies that there is a dialogue relationship between them, it indicates that the two types of objects have an interactive relationship in the same spatiotemporal scene (e.g., a pedestrian and a shopping cart are in the same frame and there is a dialogue about pushing the shopping cart, reflecting a pushing interaction relationship). In this case, the second object meets the judgment condition of having an interaction relationship with the first object, so it is identified as the third object. If the retrieval result shows that there is no dialogue between the two types of objects, even if the two types of objects are in the same frame, the second object is not identified as the third object due to the lack of evidence of interaction relationship.

[0104] As an example, in a shopping mall security detection scenario, the first object is pedestrians, and the second object is shopping carts. The specific process is as follows: Based on pedestrians and shopping carts, video frames in the video are identified to obtain the recognition results. The number and type of objects are detected for each video frame: if a frame contains only pedestrians (e.g., a pedestrian walking alone towards a shelf) or only shopping carts (e.g., a shopping cart stopped in an aisle), it is determined as the first video frame; if a frame contains multiple objects and includes both pedestrians and shopping carts (e.g., a pedestrian moving while holding a shopping cart), it is determined as the second video frame. After checking all video frames, multiple second video frames are found, so the recognition result is that there are video frames that simultaneously include pedestrians and shopping carts. Then, in response to this recognition result, the subtitles in the video are retrieved to obtain the search results. First, the playback time corresponding to the second video frame is defined (e.g., 14 minutes 20 seconds to 14 minutes 25 seconds), and only the subtitle content within this time period is retrieved. The extracted subtitles included the question, "Does this shopping cart need to be pushed to the checkout?" Yes, the preceding text was the dialogue at the checkout. Through keyword matching (including "pushing the shopping cart") and semantic analysis (the dialogue revolved around the pedestrian using the shopping cart), it was confirmed that there was a dialogue between the pedestrian and the shopping cart. Therefore, the search result was that the subtitles contained a dialogue between the two. Finally, in response to this search result, the shopping cart was identified as a third object that had an interactive relationship with the pedestrian. Because video frame recognition confirmed that the two were in the same frame, and the subtitle search verified the existence of a related dialogue, it indicated that the pedestrian and the shopping cart had an interactive relationship of pushing and using it, meeting the criteria for determining a third object.

[0105] As an example, in a smart transportation scenario, the first object is the bus driver, and the second object is the dispatch terminal. The specific process is as follows: Video frames are identified based on the bus driver and the dispatch terminal to obtain identification results. Each video frame is detected: if a frame contains only the driver (e.g., the driver adjusts the seat) or only the dispatch terminal (e.g., the terminal displays information but no user operation), it is identified as the first video frame; if a frame contains multiple objects and includes both the driver and the dispatch terminal (e.g., the driver operates the terminal), it is identified as the second video frame. After verification, a second video frame is found, and the identification result is that a video frame containing both the driver and the dispatch terminal exists. Then, in response to this identification result, video subtitles are retrieved to obtain search results. The playback time corresponding to the second video frame is defined (e.g., 7 minutes 30 seconds to 7 minutes 35 seconds), and subtitles for this time period are retrieved. The dialogue text of the dispatch terminal displaying a traffic jam ahead and requesting a detour, selecting alternative routes, is extracted. Through keyword matching (including the dispatch terminal) and semantic analysis (the dialogue revolves around the driver operating the terminal), it is confirmed that there is a dialogue between the driver and the dispatch terminal, and the search result is that there is a dialogue between the two. Finally, based on the search results, the dispatch terminal was identified as a third party with an interactive relationship with the bus driver. Since the two are displayed in the same frame and have related dialogue, it indicates an interactive relationship where the driver uses the terminal to obtain dispatch information, thus meeting the criteria for a third party.

[0106] Thus, based on the process of identifying the third object through video frame recognition and subtitle retrieval of the first and second objects, the process first confirms whether the two types of objects are in the same frame, providing a visual basis for determining the interaction relationship and avoiding the bias caused by relying on a single information source. Then, for cases where there are frames in the same frame, the corresponding subtitles are retrieved to verify whether there is a related dialogue between the two types of objects, supplementing the interaction evidence that cannot be directly presented by visual information. For example, in security detection scenarios, dialogue can be used to confirm the pushing relationship between pedestrians and shopping carts; in film and television production scenarios, dialogue can be used to verify the usage relationship between actors and sword props; and in intelligent transportation scenarios, dialogue can be used to clarify the operational relationship between drivers and dispatch terminals. This effectively avoids misjudgments caused by objects being in the same frame but without actual interaction (such as pedestrians and shopping carts being in the same frame by chance but without contact). Finally, the second object is identified as the third object only when both the frame-in-the-frame and dialogue conditions are met, significantly improving the accuracy of the interaction relationship determination and providing reliable interaction relationship data support for subsequent operations such as constructing object graph structures and updating object features, reducing the interference of invalid data on subsequent processes.

[0107] Thus, when processing each object in the video separately, the current object is first distinguished from other objects in the video. This avoids interference in subsequent interaction relationship judgment caused by the mixing of information between different objects, laying a clear foundation for accurate selection of interactive objects. Then, based on the visual information contained in the video frames (such as the action coordination between objects, the proximity of positions, and the orientation of gestures) and the textual information contained in the video subtitles (such as the semantic content describing the interaction behavior between objects and the dialogue related expressions), objects that have an interactive relationship with the current object are identified from other objects. This can combine the complementary advantages of the two types of information and avoid misjudgment or omission of interaction relationships caused by relying on only a single information source (such as relying solely on visual information, which is prone to overlooking implicit interactions without obvious actions, and relying solely on textual information, which is prone to misjudging semantically ambiguous related expressions). This ensures that the relationship between the selected interactive objects and the current object is real and accurate. This precisely defined interaction relationship provides correct association information to support subsequent updates of the current object's features based on the characteristics of the interacting object. It avoids object feature bias introduced by erroneous association information, making the updated object features more accurately reflect the object's true attributes and associated context. This provides a more reliable feature basis for speech signal classification, reduces classification bias caused by inaccurate object association information, and ultimately effectively improves the classification accuracy of speech signals.

[0108] In step 103, based on the object graph structure, the first object feature of each object is updated to obtain the second object feature of each object.

[0109] In some embodiments, the second object feature refers to a feature set formed during video processing by supplementing and improving the existing first object features of each object, based on an object graph structure. This second object feature more comprehensively reflects the object's attributes and the relationships between objects. The formation of the second object feature relies heavily on the object graph structure. The object graph structure contains the first object features of each object (as nodes) and the interaction relationships between different objects (as edges). The association information of these nodes and edges provides core data support for updating the first object features. For example, if there is a pushing interaction relationship (edge) between a pedestrian and a shopping cart in the object graph structure, the first object features of both the pedestrian and the shopping cart can be updated accordingly. The update process focuses on supplementing the missing association attribute information in the first object features. The first object feature primarily covers the object's own image, audio, and text attributes. The second object feature, by integrating interaction information from the object graph structure, adds features related to other objects. For example, in a security detection scenario, the first object feature of a pedestrian includes facial contours, concurrent dialogue, and location information. When updated to the second object feature, it adds attributes such as a pushing relationship with a shopping cart and the pushing period being from 14 minutes 20 seconds to 14 minutes 25 seconds. The second object feature of the shopping cart adds attributes such as being pushed by the pedestrian and the pushing period coinciding with the pedestrian's pushing period. The second object feature retains the core attributes of the object itself from the first object feature while adding interactive attributes related to other objects. This extends the feature information from single object attributes to the dimension of inter-object relationships, providing a more comprehensive and systematic portrayal of the object's state and role in the video. This richer feature set provides more accurate and comprehensive data support for subsequent operations such as deep object identification and semantic analysis of video content. It avoids one-sided object characterization due to lack of related information, and at the same time, it makes the interaction between object features and the object graph structure form a mutually reinforcing whole, improving the consistency and effectiveness of data in the video processing process and adapting to the needs of deep object feature analysis in various application scenarios.

[0110] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 5 , Figure 3 Step 103 shown can be performed separately for each object. Figure 7 Steps 1031 to 1032 shown are implemented.

[0111] In step 1031, target objects that have interactive relationships with the objects are determined from the object graph structure.

[0112] In some embodiments, determining the target object that has an interactive relationship with the object from the object graph structure can be achieved by: identifying the target object corresponding to the node that has a connection with the node corresponding to the object in the object graph structure as the target object that has an interactive relationship with the object.

[0113] In some embodiments, the association logic between nodes and edges in the object graph structure is explicitly defined. The object graph structure uses the characteristics of each object (such as a first object characteristic or a second object characteristic) as independent nodes. The connecting edges between nodes directly represent the interaction relationships between different objects; that is, as long as there is a connecting edge between two nodes, it means that the corresponding two objects have an actual interaction relationship in the video. The attributes of the connecting edges (such as interaction type, interaction time period) can further supplement the interaction details, but the connection relationship between nodes itself can serve as the core basis for the existence of the interaction relationship. For example, in a security detection scenario, if there is a connecting edge between the pedestrian node and the shopping cart node, it indicates that the pedestrian and the shopping cart are interacting; in a film production scenario, if there is a connecting edge between the actor node and the ancient sword prop node, it indicates that the actor and the ancient sword prop are interacting. The nodes corresponding to the objects in the object graph structure are located. First, the core features of the object whose interaction relationship needs to be determined (such as appearance in image features and identification information in text features) need to be extracted. Then, these features are matched with the object features contained in all nodes of the object graph structure to find the node whose features completely match. This node is the unique identifier of the object to be processed in the graph structure. For example, when the object to be processed is a bus in a smart transportation scenario, the core features such as the blue body and the route identification of bus 102 can be used to quickly locate the corresponding node in the object graph structure, avoiding node confusion. Identify all the connecting nodes of the located node. After determining the node corresponding to the object, traverse all other nodes in the object graph structure that have a direct connection edge with that node. These connecting nodes are the candidate nodes that have an interaction relationship with the object to be processed. There is no need to further filter unrelated nodes, because the construction process of the object graph structure has ensured that only nodes corresponding to objects with an interaction relationship will be connected. For example, the connecting nodes of the pedestrian node may include the shopping cart node and the shelf node, which correspond to different objects that interact with the pedestrian. The connecting nodes of the bus node may include the traffic light node and the pedestrian node, which also correspond to different interactive objects. Finally, the object corresponding to the connecting node is determined as the target object. Extract the object features contained in each connection node, and identify the specific object corresponding to the connection node (such as a shopping cart, shelf, traffic light, etc.) through feature matching. These objects are the target objects that have an interactive relationship with the object to be processed. If it is necessary to further clarify the interaction details, the attribute information of the connection edge can be extracted simultaneously (such as the pushing type of the connection edge between pedestrians and shopping carts, and the following instruction type of the connection edge between buses and traffic lights), but this will not affect the core determination of the target object. That is, regardless of the connection edge attributes, the object corresponding to the connection node is the target object.

[0114] In step 1032, the update model is invoked to update the first object feature of the target object based on the first object feature, thereby obtaining the second object feature of the object.

[0115] In some embodiments, an update model refers to a machine learning model specifically designed for supplementing and improving the first object features of the current object based on the first object features of the target object during video processing, thereby generating second object features of the current object. For example, it could be a graph neural network. The core function of the update model revolves around the correlation update of object features. Its input data contains two types of key information: one type is the first object features of the current object whose features need to be updated. These features cover the object's own image features, audio features matching the image features, and text features. They are the basic data for feature updates, such as the first object features of pedestrians in security detection scenarios (facial contours, synchronous dialogue, and location information) and the first object features of actors in film and television production scenarios (appearance features, synchronous dialogue, and scene annotation information). The other type is the first object features of the target object that interacts with the current object. These features contain the core attribute information of the target object and are the correlation data that triggers the update of the current object's features. For example, the first object features of a shopping cart interacting with a pedestrian (appearance, synchronous friction sound, and area annotation information) and the first object features of an ancient sword prop interacting with an actor (material features, synchronous collision sound, and prop identification information).

[0116] In some embodiments, the update model includes an i-th update layer. The above-mentioned call to the update model, based on the first object feature of the target object, updates the first object feature of the object to obtain the second object feature of the object, which can be implemented in the following way: call the first update layer, based on the first object feature of the target object, update the first object feature of the object to obtain the first intermediate feature of the object; iterate i and perform the following processing: call the i-th update layer, based on the (i-1)-th intermediate feature of the target object, update the (i-1)-th intermediate feature of the object to obtain the i-th intermediate feature of the object; determine the N-th intermediate feature of the object as the second object feature of the object.

[0117] In some embodiments, an update model including the i-th update layer (1 < i ≤ N) is called to update the first object feature of an object based on the first object feature of a target object to obtain a second object feature. The core is to gradually fuse the feature association information of the target object through a hierarchical progressive update logic to achieve the gradual improvement of the object feature. The specific implementation method is as follows: Perform the initial update operation of the first update layer to generate the first intermediate feature of the object. When calling the first update layer, two types of initial data need to be input to this layer: one is the first object feature of the object to be updated, including the image, audio, and text attributes of the object itself (such as the facial contour of a pedestrian, the synchronous dialogue, and the location information in a security detection scenario); the other is the first object feature of the target object that has an interaction relationship with the object, including the core attributes of the target object (such as the appearance form of a shopping cart, the synchronous friction sound, and the area annotation information). The first update layer has a built-in basic association analysis algorithm. By comparing the attribute relevance of the two types of features (such as the area consistency and time synchronization between a pedestrian and a shopping cart), the missing basic association attributes in the first object feature of the object (such as being in the same area as the shopping cart and having the synchronous friction sound of the shopping cart) are mined, and these basic association attributes are supplemented into the first object feature of the object to form the first intermediate feature of the object. On the basis of retaining the original core attributes of the object, this intermediate feature adds the basic association information related to the target object, laying a foundation for subsequent iterative updates.

[0118] In some embodiments, the update operation of the i-th update layer (1 < i ≤ N) is iteratively executed to gradually generate the i-th intermediate feature of the object. Each time an iteration is performed, first determine the i-th update layer to be called currently, and then input two types of data to this layer: one type is the i-1-th intermediate feature of the object (i.e., the feature obtained after the update of the previous layer, which already includes the associated attributes supplemented by the previous i-1 layers); the other type is the i-1-th intermediate feature of the target object (the target object needs to be processed synchronously according to the same hierarchical update logic, and its i-1-th intermediate feature already includes the previous i-1 layer associated attributes of the target object with other objects or the current object). The complexity of the association analysis algorithm built into the i-th update layer increases with the number of layers, and deeper association attributes can be mined: for example, the 2nd update layer can further analyze the behavioral association between the object and the target object (such as the contact between the pedestrian's hand and the shopping cart handle, and the consistency between the contact period and the shopping cart movement period) based on the 1st intermediate feature, supplement the association attributes of having contact and interaction with the shopping cart, and generate the 2nd intermediate feature; the 3rd update layer (if N ≥ 3) can mine the influencing attributes of the interaction relationship (such as the pedestrian pushing the shopping cart causing the change of the shopping cart's position, and the coincidence between the range of position change and the range of the pedestrian's movement) based on the 2nd intermediate feature, supplement the attributes of the position association change caused by pushing the shopping cart, and generate the 3rd intermediate feature. Each iteration aims to supplement deeper association information based on the features of the previous layer, so that the intermediate features of the object gradually become richer as the number of iterations increases.

[0119] As an example, refer to Figure 14 , Figure 14This is a schematic diagram of the update model provided in this application embodiment. The update model includes a first update layer 11, a second update layer 12, a third update layer 13, a fourth update layer 14, and a fifth update layer 15. The above-mentioned call to the update model, based on the first object feature of the target object, updates the first object feature of the object to obtain the second object feature of the object. This can be achieved in the following way: The first update layer 11 is called to update the first object feature of the object based on the first object feature of the target object, obtaining the first intermediate feature of the object; the second update layer 12 is called to update the object based on the first intermediate feature of the target object. The first intermediate feature of the object is updated to obtain the second intermediate feature of the object; the third update layer 13 is called to update the second intermediate feature of the object based on the second intermediate feature of the target object to obtain the third intermediate feature of the object; the fourth update layer 14 is called to update the third intermediate feature of the object based on the third intermediate feature of the target object to obtain the fourth intermediate feature of the object; the fifth update layer 15 is called to update the fourth intermediate feature of the object based on the fourth intermediate feature of the target object to obtain the fifth intermediate feature of the object; the fifth intermediate feature of the object is determined as the second object feature of the object.

[0120] In some embodiments, the Nth intermediate feature of an object is determined as the second object feature. When the iteration reaches the Nth update layer (i.e., the last layer of the update model), the Nth intermediate feature obtained after the update of this layer has been completed contains the full-dimensional association attributes from basic to deep that were supplemented during the Nth layer update process. It not only fully retains the core attributes of the object's first object feature, but also fully integrates the association information related to the target object (such as association from basic region to behavioral contact association, and then to influence result association), which fully meets the dual requirements of the second object feature for its own attributes + association attributes. Therefore, the Nth intermediate feature is directly determined as the object's second object feature. In different application scenarios, this hierarchical iterative logic can be flexibly adapted: In security detection scenarios, if N=2, the second object feature (second intermediate feature) of a pedestrian can include the associated attributes of being in the same area as a shopping cart (first layer) and touching and pushing the shopping cart (second layer); In film and television production scenarios, if N=3, the second object feature (third intermediate feature) of an actor can include the associated attributes of being in the same scene as a sword prop (first layer), holding a sword prop (second layer), and waving a sword prop to generate an action (third layer); In intelligent transportation scenarios, if N=2, the second object feature (second intermediate feature) of a bus can include the associated attributes of being at the same intersection as a traffic light (first layer) and following the traffic light instructions (second layer).

[0121] Thus, the above-mentioned update model, which includes the i-th update layer, generates the second object feature through a hierarchical iterative approach. The first update layer supplements the basic association attributes based on the first object features of the target object and the current object, and subsequent iterations of the i-th update layer... The update layer leverages the intermediate features of the previous layer to mine deeper correlation information, avoiding insufficient mining or inaccuracy caused by processing large amounts of correlation data in a single update. This allows object features to be progressively refined with iteration, extending from basic regional and temporal correlations to behavioral and impact correlations, significantly improving the completeness and accuracy of the second object's features. For example, in security detection scenarios, pedestrian features can be gradually supplemented with deeper attributes such as contact pushing and positional changes from the same area as the shopping cart; in film production scenarios, actor features can be gradually supplemented with attributes such as handheld use actions from the same scene as the sword prop. Simultaneously, the algorithm complexity of each update layer can be flexibly adapted to scenario requirements. The value of N can be adjusted according to the depth requirements of the correlation attributes, satisfying both basic correlation update needs in simple scenarios and deep correlation mining needs in complex scenarios, enhancing model adaptability. During iteration, the intermediate features of the target object and the current object are updated synchronously, ensuring that the correlation information of the two corroborates each other, avoiding correlation bias caused by lag in one-sided features, and improving the consistency of feature updates. The final generated Nth... Intermediate features, as the second set of object features, can comprehensively cover the object's own attributes and multi-dimensional related attributes, providing high-quality data support for subsequent operations such as object identification and video semantic analysis. This reduces subsequent processing errors caused by missing features. At the same time, automated hierarchical iteration requires no manual intervention, reducing operating costs and error risks, and further improving the efficiency, reliability, and practicality of the overall video processing workflow, adapting to the needs of various application scenarios.

[0122] Thus, by identifying target objects with interactive relationships from the object graph structure and updating the first object features of the target objects based on the first object features of the target objects, the contextual information associated with the current object can be effectively integrated into the object's feature representation. This allows the object's second object features to not only contain its own local information but also integrate global relational information brought about by the interactive relationships, thereby significantly enhancing the discriminative power and richness of the features. This graph-based relational modeling and feature update mechanism can effectively capture semantic dependencies, temporal relationships, or spatial connections between objects, overcoming the feature ambiguity caused by incomplete information and missing context when relying solely on the local features of a single object. This makes the updated second object features more accurately reflect the true attributes and states of the objects. Applying these second object features rich in relational information to speech signal classification tasks can provide the classification model with a more comprehensive and robust decision basis, reducing misclassification caused by one-sided or isolated feature information, thereby effectively improving the classification accuracy of speech signals and ensuring stable and excellent classification performance even in complex and variable acoustic environments.

[0123] In some embodiments, the i-th update layer includes a fusion layer and an activation layer. Calling the i-th update layer to update the (i-1)-th intermediate feature of the target object based on the (i-1)-th intermediate feature of the target object to obtain the i-th intermediate feature of the object can be implemented as follows: calling the fusion layer to fuse the (i-1)-th intermediate feature of the target object and the (i-1)-th intermediate feature of the object to obtain the (i-1)-th fused feature; and calling the activation layer to activate the (i-1)-th fused feature to obtain the i-th intermediate feature of the object.

[0124] In some embodiments, the fusion layer is invoked to fuse the (i-1)th intermediate feature of the target object with the (i-1)th intermediate feature of the object to generate the (i-1)th fused feature. The fusion layer has a built-in feature association fusion algorithm. After receiving the two types of intermediate features, it first performs standardization processing on the feature data to ensure that the attribute dimensions of the two types of features (such as the pixel dimension of image features, the frequency dimension of audio features, and the semantic dimension of text features) are consistent with the data format, thus avoiding fusion deviations caused by dimensional differences. Subsequently, the algorithm mines the association attributes related to the interaction relationship in the two types of intermediate features. For example, in a security detection scenario, the (i-1)th intermediate feature of the object contains the basic association attribute of the pedestrian and the shopping cart being in the same area, while the (i-1)th intermediate feature of the target object contains the attribute of the shopping cart's movement trajectory. The fusion layer will match the pedestrian's area attribute with the shopping cart's movement trajectory attribute to determine whether there is spatial trajectory overlap (e.g., the pedestrian's movement path is consistent with the shopping cart's movement path), and integrate this association information of the pedestrian's movement trajectory and the shopping cart's movement trajectory overlap into the original attributes of the two types of features. At the same time, the fusion layer will also perform preliminary screening of redundant information in the two types of features (e.g., duplicate timestamp annotations, unrelated background attributes), retaining the core attributes directly related to the interaction. Through the above association integration and redundancy screening, the fusion layer finally outputs the (i-1)th fused feature containing the core association information of the two types of intermediate features. This feature not only covers the key attributes of the object and the target object respectively, but also adds the integrated information of the deep association between the two.

[0125] In some embodiments, an activation layer is called to perform feature activation on the (i-1)-th fused feature to generate the i-th intermediate feature of the object. The activation layer incorporates a feature activation and optimization algorithm, whose core function is to strengthen the effective features in the (i-1)-th fused feature that are valuable for interaction relationship determination and suppress meaningless noise features. First, the algorithm assigns weights to each attribute dimension of the (i-1)-th fused feature, setting different weights according to the degree of correlation between the attribute and the interaction relationship: for example, attributes that directly reflect interactions such as trajectory coincidence and contact behavior are assigned high weights, while non-associated attributes such as background color and irrelevant ambient sounds are assigned low weights; subsequently, the weighted features are processed through a non-linear activation function to amplify the signal strength of the high-weight effective features and weaken the interference of the low-weight noise features, making the effective features more prominent; at the same time, the activation layer also performs consistency verification on the feature data. If there are attribute conflicts in the fused feature (such as trajectory coincidence and too large position distance appearing simultaneously), the conflicting attributes are corrected based on the actual video scene logic (such as if a pedestrian and a shopping cart are in contact, the position distance cannot be too large) to ensure the rationality of the features. After the activation process, the feature output by the activation layer is the i-th intermediate feature of the object. Based on the (i-1)-th intermediate feature, this feature adds newly fused and activation-optimized deep association attributes, and has stronger effective feature signals and less noise interference, being more capable of accurately reflecting the interaction association state between the object and the target object.

[0126] In some embodiments, in different application scenarios and different update layers, the processing logics of the fusion layer and the activation layer will be dynamically adjusted according to the feature depth requirements: for example, in the second update layer of the film production scenario, the fusion layer may integrate the attribute of an actor holding a sword prop and the attribute of the swinging angle of the sword prop to generate fusion information that matches the actor's hand movement and the swinging angle of the sword prop. The activation layer will then strengthen the weight of this matching attribute and suppress noise features such as irrelevant textures on the surface of the sword prop; in the third update layer of the intelligent transportation scenario, the fusion layer may integrate the attribute of a bus following traffic signal instructions and the attribute of the green light period of the traffic signal to generate fusion information that completely coincides with the bus passing period and the green light period of the traffic signal. The activation layer will then strengthen this period coincidence attribute and weaken noise features such as irrelevant advertising patterns on the bus body. Through the division of labor and cooperation between the fusion layer and the activation layer in the entire i-th update layer, feature association integration is first achieved, and then feature optimization activation is completed, ensuring that the generated i-th intermediate feature contains both comprehensive association information and high accuracy and strong effectiveness, providing a high-quality feature basis for subsequent iterative updates (if i < N) or the final generation of the second object feature (if i = N). At the same time, the design of the two-layer structure makes the feature processing logic clearer, facilitating the adjustment of the algorithm parameters of each layer according to different scenario requirements and enhancing the flexibility and adaptability of the update model.

[0127] In some embodiments, the fusion layer is the core functional module in the i-th update layer responsible for integrating the associated features of the target object and the current object. Its core function is to perform attribute association and information integration on the (i - 1)-th intermediate features of the target object and the (i - 1)-th intermediate features of the current object, generate a fusion feature containing the associated information of both, and provide basic data for subsequent feature activation. The activation layer is the core functional module in the i-th update layer responsible for screening and strengthening the effective information in the fusion feature. Its core function is to perform effective attribute screening and feature strength enhancement on the (i - 1)-th fusion feature output by the fusion layer, generate the i-th intermediate feature of the current object, and complete the feature update of the i-th update layer.

[0128] Thus, the above-mentioned method for updating the intermediate features of an object by the collaborative processing of the fusion layer and the activation layer in the i-th update layer enables the fusion layer to perform attribute association and information integration on the (i - 1)-th intermediate features of the target object and the current object, effectively mining the association relationship between the two types of features (such as the same-region attribute and synchronous sound attribute association between pedestrians and shopping carts in the security detection scenario), eliminating duplicate and redundant information and integrating effective associated content, generating a fusion feature containing the organic combination information of both, avoiding the omission of associated information caused by the isolated processing of the two types of features, and providing comprehensive basic data for subsequent feature processing; the activation layer then screens out the effective attributes directly related to the object interaction in the fusion feature (such as the shopping cart friction sound attribute) through effectiveness evaluation, eliminates irrelevant and invalid attributes (such as the shopping cart manufacturer attribute), and at the same time strengthens the feature strength of the effective attributes, avoiding interference from invalid information in subsequent processing, and significantly improving the accuracy and usability of the intermediate features; the collaborative work of the two enables the i-th intermediate feature to retain the core attributes of the object and the previous associated information, and at the same time add the effective associated attributes of the target object without redundant interference, achieving the dual effects of comprehensive integration + precise screening of features more than a single processing link, ensuring that the quality of the intermediate features after each layer update gradually improves, providing high-quality data support for subsequent iterative updates (if i < N) or generating the second object feature (if i = N), reducing subsequent processing errors caused by insufficient feature quality, and at the same time, each layer processed according to this logic can ensure the stability and reliability of the overall update model, adapting to the needs of object feature updates in various scenarios such as security detection, film and television production, and intelligent transportation, and further improving the efficiency and accuracy of the feature update link in the overall video processing process.

[0129] In some embodiments, before updating the first object feature of each object based on the object graph structure to obtain the second object feature of each object, the update model can also be trained in the following manner: Obtain an object sample graph structure with the first sample features of object samples as nodes, and call the initial update model to update each first sample feature in the object sample graph structure based on the object sample graph structure to obtain the second sample feature of each object sample; Determine the loss value of the initial update model based on the second sample feature, and train the initial update model based on the loss value to obtain the update model.

[0130] In some embodiments, an object sample graph structure required for training is obtained. The construction logic of this sample graph structure is consistent with the object graph structure in actual applications, with the first sample features of object samples as independent nodes, and the connecting edges between nodes corresponding to the interaction relationships between different object samples. Among them, the first sample features of object samples need to cover information with the same dimensions as the first object features in the actual scenario. For example, the object samples in the security detection scenario may include pedestrian samples and shopping cart samples. The first sample features of pedestrian samples include facial contour samples, concurrent dialogue samples, and regional annotation samples, and the first sample features of shopping cart samples include appearance form samples, concurrent friction sound samples, and regional annotation samples; At the same time, the clear interaction relationships between object samples (such as the pushing relationship between pedestrian samples and shopping cart samples) need to be marked in the sample graph structure to ensure that the sample data can truly simulate the object association scenarios in actual applications and provide basic data close to reality for model training.

[0131] In some embodiments, the initial update model is called to update the first sample features in the object sample graph structure to obtain the second sample feature of each object sample. The structure of the initial update model is the same as that of the update model for final application, including N update layers, and each update layer is provided with a fusion layer and an activation layer. During the call process, first determine the target sample corresponding to each object sample from the object sample graph structure (that is, other object samples that have an interaction relationship with this object sample, screened through the node connection edges); then process according to the hierarchical iteration logic: Call the first update layer to fuse and activate the first sample feature of the object sample based on the first sample feature of the target sample to obtain the first sample intermediate feature of the object sample; Subsequently, iteratively call the i-th update layer (1 < i ≤ N), and fuse (the fusion layer integrates the associated attributes of the two types of sample intermediate features) and activate (the activation layer screens out the effective attributes and strengthens them) the i-1 sample intermediate feature of the object sample based on the i-1 sample intermediate feature of the target sample to obtain the i sample intermediate feature of the object sample; Finally, determine the N sample intermediate feature as the second sample feature of the object sample to complete the update operation of the sample features by the initial model.

[0132] In some embodiments, the loss value of the initial update model is determined based on the second sample features, and the model is optimized based on the loss value to obtain the updated model. The calculation of the loss value relies on the preset true labels (i.e., the standard second features that the object sample should have. These labels need to be determined in combination with the actual interaction scenario and attribute association logic of the object sample. For example, the standard second features of pedestrian samples should include attributes such as the push association with shopping cart samples and the push time being consistent with the shopping cart movement time). The loss value is calculated by comparing the differences between the second sample features output by the initial model and the true labels (such as the number of missing attributes and the degree of attribute error) using a preset loss function (such as the mean squared error function and the cross-entropy function). If the loss value is higher than the preset threshold, it indicates that the sample features output by the initial model deviate significantly from the actual situation. The model parameters (such as the association analysis weight of the fusion layer and the effectiveness evaluation threshold of the activation layer) need to be adjusted in reverse based on the loss value. The process of calling the initial model to update sample features, calculating the loss value, and adjusting the model parameters is repeated until the loss value is lower than the preset threshold. At this point, the initial model has the ability to accurately update the object features and is determined as the final applicable update model.

[0133] In some embodiments, when determining the loss value of the initial update model based on the second sample features, a specific implementation of triplet loss is adopted. The core is to construct a triplet containing anchor samples, positive samples, and negative samples, calculate the distance difference of the second sample features between different samples, and then construct the loss value based on a preset interval to quantify the deviation between the model output features and the true correlation between the samples. The specific implementation process is described in detail below.

[0134] In some embodiments, triplet samples are selected from the object sample set used for training. Triplet samples include three types of samples: the first type is anchor samples, which are any object sample with generated second sample features (e.g., a pedestrian sample in a security detection scenario), and their second sample features are denoted as anchor features; the second type is positive samples, which are object samples that have a real interaction relationship with the anchor samples (e.g., a shopping cart sample corresponding to the aforementioned pedestrian sample). This sample must have a connection edge with the anchor sample in the object sample graph structure, and its second sample features are denoted as positive sample features; the third type is negative samples, which are object samples that have no real interaction relationship with the anchor samples (e.g., another shelf sample corresponding to the aforementioned pedestrian sample). This sample has no connection edge with the anchor samples in the object sample graph structure, and its second sample features are denoted as negative sample features. During the selection process, it must be ensured that the anchor samples, positive samples, and negative samples of each triplet come from the same object sample set of the application scenario, and that the interaction relationship between the positive samples and the anchor samples, and the lack of interaction relationship between the negative samples and the anchor samples, are consistent with the annotations in the sample graph structure, avoiding the impact of sample selection bias on the accuracy of loss value calculation.

[0135] In some embodiments, the distances between anchor features and positive sample features, and between anchor features and negative sample features, are calculated. The distance calculation is based on the attribute dimensions of the second sample features (such as image feature dimension, audio feature dimension, and text association feature dimension), and uses a preset distance metric (such as Euclidean distance or cosine distance) to calculate two types of distances: the first type is the distance between anchor features and positive sample features (denoted as positive distance), which characterizes the similarity of features between sample pairs with interactive relationships. Theoretically, because there is a real correlation between the two, the positive distance should be small. The second type is the distance between anchor features and negative sample features (denoted as negative distance), which characterizes the similarity of features between sample pairs without interactive relationships. Theoretically, because there is no correlation between the two, the negative distance should be large. During the calculation process, each attribute dimension of the second sample features needs to be standardized to ensure that the attribute weights of different dimensions are consistent, avoiding excessive influence of excessively large or small attribute values ​​on the distance calculation results. For example, pixel values ​​in image features, decibel values ​​in audio features, and association scores in text features are all standardized to the same numerical range before distance calculation.

[0136] In some embodiments, a triplet loss value is constructed based on positive distance, negative distance, and a preset interval. First, a preset interval is set (this interval is a fixed value determined in advance based on the sample feature distribution and scenario requirements, used to ensure that the feature distance between sample pairs with no interaction relationship is sufficiently different from that between sample pairs with interaction relationship; for example, in a security detection scenario, the interval can be set to 0.5). Then, the loss value is constructed according to the core logic of triplet loss: when the difference between the negative distance and the positive distance is greater than the preset interval, it indicates that the features output by the model can effectively distinguish between sample pairs with and without interaction relationship, and the loss value is 0; when the difference between the negative distance and the positive distance is less than or equal to the preset interval, it indicates that the features output by the model have failed to sufficiently distinguish between the two types of sample pairs, and a loss value needs to be calculated to guide model optimization. In this case, the loss value is the preset interval plus the positive distance minus the negative distance. After calculating the loss value of a single triplet, it is necessary to traverse all constructed triplets in the training set, calculate the loss value of each triplet, and take the average value to obtain the overall triplet loss value of the initial updated model in the current training round. This average value can comprehensively reflect the model's ability to distinguish the association relationship of all sample pairs.

[0137] In some embodiments, this implementation can be flexibly adapted to different application scenarios: In film and television production scenarios, anchor samples are actor samples, positive samples are sword prop samples, and negative samples are lighting equipment samples (when there is no interaction). The loss value is constructed by calculating the distance between the second sample features of the three types of samples and a preset interval; In intelligent transportation scenarios, anchor samples are bus samples, positive samples are traffic light samples, and negative samples are private car samples (when there is no interaction), and the loss value is calculated according to the same logic. The entire process quantifies the model performance through the feature distance difference of triplet samples, which can effectively guide the initial update model to learn the rule that the features of samples with interactive relationships are more similar and the features of samples without interactive relationships are more different. This avoids the model misjudging the relationship between samples and provides accurate loss feedback for subsequent adjustment of model parameters based on the loss value (such as the association weight of the fusion layer and the screening threshold of the activation layer). This ensures that the finally trained update model can stably and accurately update the object features in practical applications, improving the reliability and accuracy of the feature update link in the overall technical solution.

[0138] Thus, by acquiring the object sample graph structure, calling the initial update model to generate second sample features, and optimizing the model based on the loss value to obtain the training method of the updated model, the object sample graph structure uses the first sample features of the object samples as nodes and retains the interaction relationship between samples. This is consistent with the object graph structure construction logic in actual applications, providing data support for model training that closely resembles real-world scenarios and avoiding insufficient model adaptability due to the disconnect between sample data and actual scenarios. Calling the initial update model to update the first sample features according to the hierarchical iterative logic to obtain the second sample features simulates the complete feature update process in actual applications, enabling the model to grasp the rules of feature association updates during the training phase and reducing adaptation errors in actual applications. Determining the loss value based on the second sample features (such as using triplet loss to quantify the deviation between features and the true association) and optimizing the model can refine the model through loss feedback. The model parameters (such as the association weights of the fusion layer and the screening thresholds of the activation layer) are precisely adjusted to ensure that the model gradually acquires the ability to distinguish the relationship between samples and accurately update features, avoiding feature update deviations caused by unreasonable model parameters. The entire training process forms a closed loop from data preparation to model optimization. The updated model obtained by training can stably and accurately update the first object features of objects based on the object graph structure in practical applications, generating complete and accurate second object features, providing high-quality data support for subsequent operations such as object identification and video semantic analysis. At the same time, this training method can be flexibly adapted to various scenarios such as security detection, film and television production, and intelligent transportation. By replacing the object sample graph structure of the corresponding scenario, an updated model adapted to the scenario can be trained, significantly improving the model's versatility and practicality, and further enhancing the reliability and efficiency of the overall technical solution in different scenarios.

[0139] In step 104, for each object, the object is identified based on the object's second object feature to obtain the object's object identity.

[0140] In some embodiments, object identity refers to the information that uniquely identifies the essential attributes or category of an object, determined during video processing based on its second object features through feature matching and attribute verification. It is a clear definition of what the object is or to which category it belongs in the video scene, with the determination of identity based primarily on the object's second object features. These second object features include both the object's core attributes (such as appearance and texture details in image features, sound frequency and rhythm in audio features, and identification information in text features) and its association attributes with other target objects (such as interaction type, interaction time period, and interaction scene). These attributes together constitute the complete data foundation for identity recognition. For example, in a security detection scenario, a pedestrian's second object features include facial contours, clothing features (inherent attributes), and the interaction relationship with a shopping cart, as well as the interaction time period (association attributes). These features can serve as the basis for identifying pedestrians. In a smart transportation scenario, a bus's second object features include its body color, route markings (inherent attributes), and the relationship with traffic lights, as well as the travel time period (association attributes), which can serve as the basis for identifying the bus. The representation of object identity varies depending on the type of object in the application scenario: In scenarios involving specific individuals, object identity can be represented as an individual-specific identifier (such as the identity number of a pedestrian or the name information corresponding to facial features in a security detection scenario); in scenarios involving category attribution, object identity can be represented as a clear category label (such as the category identifier of ancient sword props in a film and television production scenario or the route category of bus route 102 in a smart transportation scenario); in scenarios involving functional attributes, object identity can be represented as functional definition information (such as the functional identifier of promotional shelves in a shopping mall scenario or the functional category of a handling robot in a factory scenario).

[0141] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 6 , Figure 3 Step 104 shown can be executed Figure 8 Steps 1041A to 1043A shown are implemented.

[0142] In step 1041A, a preset mapping relationship between multiple preset object identities and multiple preset object features is obtained, wherein the preset mapping relationship includes a preset index entry between each preset object identity and the corresponding preset object feature.

[0143] In some embodiments, the source and scope of preset object identities and preset object characteristics are clearly defined. Preset object identities need to cover all object identity types that may appear in the target application scenario, and their representation is consistent with the definition logic of object identities in the actual scenario: In scenarios involving individual identification (such as security detection), preset object identities may include the identity number and name identifier of registered pedestrians; in scenarios involving category attribution (such as film and television production), preset object identities may include category labels such as ancient sword props, modern gun props, and main lighting equipment; in scenarios involving functional attributes (such as intelligent transportation), preset object identities may include functional identifiers such as the pedestrian crossing sign at intersection A of bus route 101. Preset object features are a set of standard features corresponding to each preset object identity. They must include attributes of the same dimension as the second object features (its own core attributes and related attributes), and the attribute values ​​must conform to the standard feature performance of the identity object. For example, when the preset object identity is bus route 101, the corresponding preset object features must include the '101' route logo on the blue bus body (its own core attribute) and the standard features that it follows the instructions of traffic lights and that the regular travel time is 6:00-22:00 (related attribute). When the preset object identity is an ancient sword prop, the corresponding preset object features must include the standard features that it is made of iron, the sword is 80cm long (its own core attribute), and that it is used by actors and the usage scene is an ancient costume fight scene (related attribute).

[0144] In some embodiments, preset index entries are constructed in the preset mapping relationship. Each preset index entry corresponds to an association record of a preset object identity and a set of preset object features. The entry must clearly label two types of key information: one is the preset object identity identifier, using a representation consistent with actual identity recognition (such as identity number, category label, function identifier) ​​to ensure accurate positioning during subsequent comparisons; the other is a detailed list of all preset object features corresponding to that identity, categorized by its core attributes and associated attributes. Each attribute must include a specific feature value or feature range (e.g., vehicle color: blue; permitted hours: 6:00-22:00; material: iron) to avoid comparison errors due to ambiguous feature descriptions. Simultaneously, to improve subsequent retrieval efficiency, each preset index entry must be assigned a unique index number. The numbering rules can be compiled according to application scenarios (e.g., security detection scenarios use AJ + numerical number, intelligent transportation scenarios use JT + numerical number), enabling the index entry to have a quickly locatable identifier in the mapping relationship. All preset index entries are integrated to form a complete preset mapping relationship. The integration process requires first classifying and organizing the preset index entries for each scenario, and dividing the mapping relationship into sub-libraries according to the type of preset object identity (individual identifier, category label, function identifier) ​​or application scenario module (such as a sub-library for security detection objects, a sub-library for film and television production props, and a sub-library for intelligent transportation facilities), so as to facilitate subsequent calling of the corresponding sub-libraries according to the actual application scenario; then establishing the association index between the sub-libraries. If different sub-libraries have cross-related preset object identities (such as pedestrian identities may appear in both security detection and intelligent transportation scenarios), the association index must be used to ensure that the corresponding preset object characteristics remain consistent in different sub-libraries; finally, the completeness and accuracy of the integrated preset mapping relationship are verified to check for problems such as missing preset object identities, missing preset object characteristics or incorrect attribute values, and duplicate index entries. After the verification is passed, the final preset mapping relationship that can be used for identity recognition and comparison is formed.

[0145] In step 1042A, the preset object feature that is closest to the second object feature is queried from the preset mapping relationship, and the preset index entry where the preset object feature is located is determined as the target index entry of the object.

[0146] In some embodiments, attribute dimension data of all preset object features in the preset mapping relationship between the second object features and the preset object features are extracted. First, the complete attribute composition of the second object features needs to be clearly defined, and the specific feature values ​​of each dimension are decomposed according to its own core attributes and related attributes (e.g., in a security detection scenario, the second object feature of a pedestrian is decomposed into facial contour: ellipse, clothing: blue coat (its own core attribute), interactive object: shopping cart, and interaction time: 14:20-14:25 (related attribute)). Then, all preset index entries in the preset mapping relationship are traversed, and the corresponding attribute dimension data of the preset object features in each entry is extracted to ensure that the attribute dimensions of the two are completely matched (e.g., the preset object features also include dimensions such as facial contour, clothing, interactive object, and interaction time), avoiding distance calculation deviations due to missing dimensions. If the preset mapping relationship is divided into sub-libraries according to scenarios, the corresponding sub-library needs to be called first according to the application scenario to which the current object belongs (e.g., security detection), and only the attribute data of the preset object features in the sub-library is extracted, reducing the amount of irrelevant data processing and improving query efficiency.

[0147] In some embodiments, the comprehensive distance between the second object feature and each preset object feature is calculated. The distance calculation requires an appropriate measurement method for the feature type of different attribute dimensions: for numerical attributes (e.g., sword length 80cm, travel time 6:00-22:00), Euclidean distance is used to calculate the numerical difference between the two (e.g., the sword length in the second object feature is 78cm, and the difference between the preset 80cm is calculated by the square root of the square difference); for categorical attributes (e.g., car body color blue, material iron), Hamming distance is used to determine if the attributes are consistent (0 for the same attributes, 1 for different attributes); for relational attributes (e.g., interaction with a shopping cart and interaction with an actor holding a hand), semantic similarity is used to determine the degree of matching of the relational type (e.g., pushing and pulling have low semantic similarity, resulting in a large distance; holding and gripping have high semantic similarity, resulting in a small distance). After calculating the distances in each dimension, the distances in each dimension are weighted and summed according to preset weights (e.g., 60% for the core attribute and 40% for the relational attribute) to obtain the comprehensive distance between the second object feature and the preset object feature. The smaller the comprehensive distance, the higher the similarity between the two.

[0148] In some embodiments, the preset object feature with the smallest comprehensive distance is selected and the target index entry is determined. The comprehensive distances corresponding to all preset object features are traversed, and the feature with the smallest comprehensive distance is found. The preset object feature corresponding to this distance is the feature most similar to the second object feature. Then, the preset index entry to which this preset object feature belongs is located (by matching the feature details within the preset index entry with the selected preset object feature), and this index entry is determined as the target index entry for the object. If multiple preset object features have the same comprehensive distance and are all minimum values ​​(e.g., the facial contours and clothing features of two preset pedestrian identities are very similar to the second object feature), the attribute dimensions need to be further refined (e.g., adding sub-attributes such as hairstyle and accessories) to recalculate the distance, or the environmental attributes of the object's scene (e.g., the first floor of a shopping mall, intersection A) are combined to select preset object features that are more suitable for the scene, ultimately determining a unique target index entry.

[0149] In step 1043A, the preset object identity in the target index entry is determined as the object identity of the object.

[0150] As an example, a preset mapping relationship is used. The preset index entries of this mapping relationship cover commonly used props and equipment in film and television, such as index entry 1 (identity: ancient wooden sword prop, characteristics: wooden material, sword length 75cm, hilt wrapped with brown cloth strips, often interacted with by actors, used in ancient costume fight scenes), index entry 2 (identity: main lighting equipment A, characteristics: metal casing, adjustable illumination angle, often interacts with cameras, used in the center of the studio), and index entry 3 (identity: modern gun prop, characteristics: plastic material, simulated black appearance, no interaction records, stored in the prop room). Entries are stored in the film and television production sub-database according to props and equipment, labeled with index numbers (e.g., YZ001, YZ002, YZ003). The target index entry is queried from the preset mapping relationship. The second object of a certain object has the characteristics of being wooden material, having a sword length of 75cm, a hilt wrapped with brown cloth strips, and interacting with actors in ancient costume fight scenes. After calling the film and television production sub-library, the comprehensive distance between this feature and each preset feature is calculated: it perfectly matches the material, size, and interactive scene dimensions of index entry 1, resulting in the smallest comprehensive distance; the comprehensive distances with index entry 2 (no sword blade feature, inconsistent interactive object) and index entry 3 (material is plastic, but appearance and interactive attributes differ greatly) are significantly larger; therefore, index entry 1 is determined as the target index entry. The object identity of the object is determined. The preset object identity (ancient wooden sword prop) of index entry 1 is extracted and used as the object identity of this object.

[0151] As an example, a preset mapping relationship is retrieved. The preset index entries for this mapping relationship include common road objects, such as index entry 1 (identity: bus route 102, characteristics: red body, white route 102 marking on the body, length 10 meters, frequently interacts with traffic lights, regular operating hours 6:00-22:00), index entry 2 (identity: traffic light at intersection A, characteristics: metal pole support, three-color light group, frequently interacts with buses, location at a city center intersection), and index entry 3 (identity: pedestrian crossing sign, characteristics: blue background with white text, marked "pedestrian crossing," no interaction records, location on both sides of intersection A). Entries are stored in the intelligent transportation sub-database according to vehicle traffic facilities, labeled with index numbers (e.g., JT001, JT002, JT003). The target index entry is then queried from the preset mapping relationship. The second characteristic of a certain object is a red vehicle body, a white route marker indicating "Route 102", a length of 10 meters, and an interaction with the traffic light at intersection A at 6:30. After calling the intelligent transportation sub-database, the comprehensive distance is calculated: it perfectly matches index entry 1 in terms of vehicle body color, route marker, and interaction relationship dimensions, resulting in the smallest comprehensive distance; it has the largest comprehensive distance compared to index entry 2 (no vehicle body characteristics, and the interaction role is that of the instructor rather than the follower) and index entry 3 (complete mismatch in feature dimensions); therefore, index entry 1 is determined as the target index entry. The object identity of the object is determined. The preset object identity (Route 102 bus) of index entry 1 is extracted and used as the object identity of this object.

[0152] Thus, by acquiring the preset mapping relationship, querying the target index entry, and determining the object's identity, the preset mapping relationship stores preset index entries containing preset object identities and corresponding features according to scenarios. This provides standardized and scenario-appropriate comparison criteria for identity recognition, avoiding recognition confusion caused by a lack of unified standards. For example, in security detection scenarios, entries categorized by personnel and items can accurately adapt to the object recognition needs of shopping malls, while in film production scenarios, entries categorized by props and equipment can efficiently support on-set object recognition. By querying the preset mapping relationship for the preset object feature closest to the second object's feature and determining the target index entry, accurate filtering can be achieved through multi-dimensional feature distance calculation and comprehensive comparison. The system identifies the most matching features and corresponding entries, avoiding misidentification caused by single-dimensional feature comparison. For example, in smart transportation scenarios, it combines vehicle color, route markings, and interactive relationships to calculate distances from multiple dimensions, accurately identifying the bus. It determines the preset object identity in the target index entry as the object identity, directly relying on standardized entries to complete identity mapping without additional complex calculations, significantly improving identity recognition efficiency. The entire process is tightly integrated, ensuring the accuracy and reliability of identity recognition, while also flexibly adapting to various scenarios such as security detection, film and television production, and smart transportation through categorized and stored preset mapping relationships. At the same time, the automated query and matching process reduces manual intervention, lowering operational errors and costs.

[0153] In some embodiments, see Figure 9 , Figure 9 This is a flowchart illustrating the identity recognition method provided in the embodiments of this application. Figure 7 , Figure 3 Step 105 shown can be executed Figure 9 Steps 1041B to 1042B shown are implemented.

[0154] In step 1041B, the identity recognition model is invoked to perform identity recognition on the object based on the second object feature of the object, thereby obtaining the probability that the object corresponds to each preset object identity.

[0155] In some embodiments, the input data and preprocessing logic of the identity recognition model are clearly defined. The core input data of the model is the second object feature of the object, which includes the object's core attributes (such as appearance details in image features, sound parameters in audio features, and identification information in text features) and related attributes (such as interaction relationship type, interaction time period, and interaction scene). Before input, the second object feature needs to be standardized and preprocessed: the attribute data of different dimensions are converted into a unified format that the model can recognize (such as converting the pixel matrix of image features into feature vectors and the descriptive information of text features into encoding vectors). At the same time, numerical attributes (such as the sword length of 75cm and the passage time of 6:00-22:00) are normalized to ensure that the numerical range of each attribute dimension is consistent and to avoid excessive influence of the probability calculation by an excessively large value of a certain attribute dimension. In addition, the model needs to load a preset set of object identities and the standard features corresponding to each preset object identity (consistent with the preset object features in the preset mapping relationship) as a comparison benchmark for probability calculation. The feature matching degree is calculated based on the built-in algorithm of the model and converted into probability. The model first uses a feature extraction module to perform deep feature mining on the preprocessed features of the second object and the standard features of each preset object identity, extracting key features with discriminative power from the two types of features (such as facial key point features of pedestrians in security detection scenarios, and texture features of bus route markings in intelligent transportation scenarios). Then, a similarity calculation module calculates the matching degree between the features of the second object and each preset standard feature. The calculation process needs to incorporate attribute dimension weights (e.g., the weight of core attributes is higher than that of associated attributes to ensure the core features dominate the matching degree). Cosine similarity, Pearson correlation coefficient, and other methods are used to quantify the matching degree. The matching degree ranges from 0 to 1; the closer the value is to 1, the more similar the features are. Subsequently, a probability transformation module converts the matching degree into the probability of the corresponding preset object identity. The transformation logic requires that the sum of the probabilities of all preset object identities is 1: first, all matching degrees are normalized (each matching degree is divided by the sum of all matching degrees), and then a softmax function is used (mapping the normalized matching degree to 0 to 1). The probability distribution is further optimized by considering intervals (where the sum is 1) to ensure statistical significance. For example, if a second object's feature has a match of 0.8 with a preset identity A, a match of 0.2 with identity B, and a match of 0 with other identities, after normalization and softmax processing, the probability for identity A is 0.8, the probability for identity B is 0.2, and the probability for other identities is 0. The output is the probability result for each preset object identity.The model output is presented as a probability list. Each entry in the list includes a preset object identity identifier (e.g., pedestrian registration in a shopping mall, ancient wooden sword prop, bus route 102) and a corresponding probability value. The probability values ​​are accurate to several decimal places (e.g., four decimal places), facilitating an intuitive assessment of the matching probability of each identity. Simultaneously, the model can sort the probability results in descending order, highlighting the preset object identity with the highest matching probability, providing a priority reference for determining the final object identity. For example, in a film production scenario, the object's second object feature has the highest matching degree with the ancient wooden sword prop, corresponding to a probability of 0.92, while the probability with the modern gun prop is 0.05, and the probability with the main lighting equipment is 0.03. The output probability list clearly presents this distribution, clearly indicating that the ancient wooden sword prop is the most likely object identity.

[0156] As an example, see Figure 15 , Figure 15 This is a schematic diagram of the structure of the identity recognition model provided in this application embodiment. The identity recognition model provided in this application embodiment may include an encoding layer 31 and a decoding layer 32. The above-mentioned calling of the identity recognition model, based on the second object feature of the object, performs identity recognition on the object and obtains the probability of the object corresponding to each preset object identity, can be implemented in the following way: calling the encoding layer 31 to encode the second object feature of the object to obtain object encoding features; calling the decoding layer 32 to perform identity recognition on the object based on the object encoding features and obtain the probability of the object corresponding to each preset object identity.

[0157] In step 1042B, the preset object identity with the highest probability is determined as the object identity of the object.

[0158] As an example, the identity recognition model is called to calculate the probability of an object corresponding to each preset object identity. A certain object's second object feature is that it is made of wood, has a sword length of 75cm, a hilt wrapped with brown cloth, and interacts with an actor wielding it in a period drama fight scene. After the model inputs this feature, the preprocessing stage converts the material parameters into a material feature vector, normalizes the sword length to the 0-1 range, and converts the interaction scene into a scene encoding vector; it also loads a preset set of film and television prop object identities (including ancient wooden sword props, ancient iron sword props, modern simulated sword props, decorative sword ornaments, etc.) and their corresponding standard features. The model extracts key features such as wood texture, sword length ratio, and hilt wrapping details. The matching degree is calculated using Pearson correlation coefficient: the matching degree with the standard features of an ancient wooden sword prop (wood, 75cm blade, brown cloth hilt, suitable for ancient costume fighting interactions) is 0.93; with an ancient iron sword prop (iron, 80cm blade) it is 0.15; with a modern simulated sword prop (plastic material, no cloth hilt) it is 0.09; and with a decorative sword ornament (no interaction records) it is 0.03. After normalization and softmax function processing, the output probability list is: ancient wooden sword prop 0.85, ancient iron sword prop 0.12, modern simulated sword prop 0.02, decorative sword ornament 0.01, and all other identities have a probability of 0. The preset object identity with the highest probability is determined as the object identity. The probability of 0.85 corresponding to the ancient wooden sword prop is the maximum value; therefore, the ancient wooden sword prop is determined as the object identity for this object.

[0159] As an example, the identity recognition model is called to calculate the probability of an object corresponding to each preset object identity. A certain object's second characteristic is a red vehicle body, a white route number 102 marked on the body, a length of 10 meters, and an interaction with traffic light A at 6:30 AM. After inputting this characteristic into the model, the preprocessing stage converts the vehicle body color into an RGB feature vector, the route number marking into a text encoding vector, and normalizes the vehicle length to the 0-1 range; it then loads the smart transportation preset object identity set (including bus route 102, bus route 101, private cars, taxis, etc.) and their corresponding standard features. The model extracts key features such as vehicle body color saturation and route sign font texture, and calculates the matching degree using cosine similarity: the matching degree with the standard features of bus route 102 (red body, route 102 sign, 10-meter length, high frequency at intersection A following traffic lights) is 0.96; the matching degree with bus route 101 (green body, route 101 sign) is 0.10; the matching degree with private cars (no route sign, random interaction) is 0.07; and the matching degree with taxis (blue body, no route sign) is 0.04. After normalization and softmax function processing, the output probability list is: bus route 102 0.88, bus route 101 0.09, private car 0.02, taxi 0.01, and the probabilities of other identities are all 0. Next, the preset object identity with the highest probability is determined as the object identity. The probability of bus route 102, 0.88, is the maximum value, therefore bus route 102 is determined as the object identity of this object.

[0160] Thus, by calling the identity recognition model to obtain the probability of each preset object identity corresponding to the object based on the features of the second object, and determining the preset object identity with the highest probability as the object identity, the identity recognition model can effectively mine the core information related to identity in the features through standardized preprocessing of the features of the second object (such as converting facial contours into feature vectors and normalizing color parameters) and key feature extraction (such as facial key points and material texture), avoiding recognition bias caused by chaotic original feature dimensions or omission of key information. The matching degree is calculated using methods such as cosine similarity and Pearson correlation coefficient, and the probability is obtained after normalization and softmax function processing. This quantitatively reflects the degree of matching between the object and each preset identity, upgrading identity recognition from qualitative judgment to quantitative analysis. Compared with simple feature comparison, this is more scientific, and the probability distribution can clearly present the matching confidence, reducing the risk of misjudgment. Determining the identity with the highest probability as the object identity eliminates the need for complex manual decision-making, quickly locking in the most likely identity, significantly improving recognition efficiency, and can be flexibly adapted to different scenarios. In security detection scenarios, probability calculations based on facial and interactive features ensure accurate identification of personnel. In film and television production scenarios, probability calculations based on material and scene interaction features enable accurate identification of props. In intelligent transportation scenarios, probability calculations based on vehicle body and route marking features complete vehicle identification. The entire process is highly automated, reducing errors caused by human intervention. At the same time, the probabilistic results provide quantitative basis for subsequent possible identity verification, further enhancing the reliability and practicality of the identity recognition process. This lays a precise foundation for subsequent operations such as behavior detection and scene management based on clearly identified objects, improving the efficiency and adaptability of the overall video processing workflow.

[0161] Thus, by extracting the first object feature from each object in the video, an initial foundation of the object's own attributes is provided for identity recognition. An object graph structure is constructed using the first object feature as nodes and the interaction relationships between different objects as edges. This systematically links the association information of each object in the video, breaking the isolation of a single object feature and providing comprehensive support for subsequent feature updates. Based on the object graph structure, the first object feature of each object is updated to obtain the second object feature. This second object feature not only retains the object's own core attributes but also incorporates the interaction and association attributes with other objects, compensating for the limitation of the first object feature containing only its own attributes. This makes the obtained second object feature more discriminative. By performing identity recognition based on the second object feature, relying on more complete and discriminative feature data, misjudgments caused by insufficient feature dimensions or incomplete information are reduced. By supplementing the key information required for identity recognition, the impact of problems such as confusion between similar objects and incomplete feature information on the recognition results is effectively reduced, thereby effectively improving the accuracy of identity recognition.

[0162] The following will describe an exemplary application of the embodiments of this application in a real-world identity recognition scenario.

[0163] This application introduces multimodal information. By binding a person's visual features with their name in the subtitles and their voice (voiceprint), this application establishes a more stable identity identifier. Clothes can be changed daily, but names and voices typically remain constant within a work. This solves the problem of long-term identity consistency. Contextual awareness is also introduced. By analyzing the co-occurrence relationships between characters (who always appears together), this application provides the system with a second layer of judgment beyond visual perception. Even if two players wearing team uniforms look very similar, but one always partners with forward A and the other with defender B, the system can accurately distinguish them using this social information. Previously overlooked information is recovered. Subtitles, voices, and character relationships—data that was previously ignored—now become the core basis for identity recognition and reasoning in this application. This application, through multimodal binding and contextual relationship reasoning, solves the fundamental shortcomings of existing pure visual technologies in identity consistency and distinguishing similar characters. This application can directly solve the long-standing problem of system face blindness when celebrities or actors change clothes. At the same time, the search has become more accurate, allowing users to precisely find all clips of two actors in the same frame, greatly enhancing the user's interactive experience.

[0164] To address the issues of unstable identities and difficulty in distinguishing similar individuals in related technologies, this application's embodiments construct stable identity profiles by fusing multimodal information from videos and using relationships between individuals in a scene for reasoning, thereby achieving more accurate and robust person recognition.

[0165] In some embodiments, see Figure 10 , Figure 10This is a schematic diagram illustrating the principle of multimodal feature extraction provided in this application embodiment. From the original video, an information-rich identity profile containing multiple clues is established for each person (i.e., the object described above), referred to as a multimodal trajectory (i.e., the second object feature described above). In step 201, the original video is input. In steps 202, 203, and 204, through parallel processing of visual processing (detection), text processing (subtitle recognition), and audio processing (voiceprint recognition), visual trajectories (i.e., image features described above), text names (i.e., the first text features described above), and voiceprint features (i.e., the first audio features described above) are obtained. The input video is processed using existing person detection and detection algorithms to obtain the pure visual trajectory of each person in the video. A trajectory V_i is the image slice sequence of person i on consecutive frames t: V_i = {img_i^1, img_i^2, ..., img_i^T}. Multimodal information extraction and alignment: Text information: Subtitle text is extracted through Optical Character Recognition (OCR) or direct parsing of subtitle files, and Named Entity Recognition (NER) technology is used to identify the names N_i and their timestamps t_n. Audio information: The video audio track is processed, and audio is segmented into segments by different speakers using voiceprint recognition technology. A voiceprint feature vector A_i is extracted for each segment, and its corresponding timestamp t_a is recorded. The key is to bind the above information to the corresponding visual trajectory. In step 205, based on timestamp alignment and binding: if a person's name N_i appears in the subtitles near time t, and a voiceprint A_i is detected in the audio track, while a person i in the image is detected as speaking (achieved through mouth movement analysis or sound source localization), a multimodal trajectory T_i is obtained based on timestamp alignment and binding. Its mathematical representation is a tuple: T_i = (V_i, N_i, A_i), where V_i is the visual image sequence, N_i is the identified text name, and A_i is the voiceprint feature vector. This structure forms the data basis for subsequent processing in this application, and it is much more stable than a simple visual trajectory.

[0166] In some embodiments, see Figure 11 , Figure 11This is a schematic diagram illustrating the principle of scene awareness provided in this application embodiment. In step 301, each modality is encoded separately. Single-modal feature extraction involves designing an encoder for each modality: (each modality is encoded separately) Visual encoder E_v: A CNN or ViT-based model that encodes the visual sequence V_i into an aggregated visual feature f_v_i. Text encoder E_n: A BERT-based model that encodes the name N_i into a text feature f_n_i. Audio encoder E_a: A voiceprint recognition model that encodes the voiceprint A_i into an audio feature f_a_i. In step 302, preliminary feature fusion is performed: the features from the three modalities are fused to obtain a preliminary identity feature f_fused_i that does not contain scene information. A simple fusion approach is to concatenate the features and then reduce the dimensionality through a fully connected layer (FC): f_fused_i = FC(Concat(f_v_i, f_n_i, f_a_i)); this f_fused_i is already much more robust than the pure visual features f_v_i.

[0167] In some embodiments, see Figure 11 In step 303, a scene task relationship graph is constructed: the video scene is analyzed, and all characters appearing in the same scene are constructed into an undirected graph (i.e., the object graph structure described above) G = (U, E). Here, the node set U represents all characters in the scene, and u_i ∈ U represents character i. If character i and character j co-occur or interact in the scene, an edge (u_i, u_j) ∈ E is connected between them. In step 304, graph neural network (GNN) inference (context awareness) is performed: the preliminary feature f_fused_i obtained in the previous step is used as the initial feature of each node u_i in the graph. Then, this embodiment of the application applies a graph neural network model (i.e., the update model described above) to update the features of each node by propagating and aggregating information from neighboring nodes on the graph structure. Each layer update of a GNN can be represented as: h_i^(k+1) = σ(W^(k) * h_i^(k) + Σ_{j∈N(i)} (U^(k) * h_j^(k))), where: h_i^(k) is the feature vector of node i in the k-th layer, with the initial feature h_i^(0) = f_fused_i. N(i) is the set of neighboring nodes of node i. W^(k) and U^(k) are the learnable weight matrices of the k-th layer. σ is a non-linear activation function, such as ReLU. This process simulates human reasoning: to know who a person is, look at who they are with. After propagation through K layers of GNN, the features of each node are incorporated with information from its social circle. The final output h_i^(K) of the GNN is the final, context-aware identity feature f_final_i.

[0168] In some embodiments, see Figure 12 , Figure 12 This is a schematic diagram illustrating the principle of model training and retrieval provided in this application embodiment. The training objective is to optimize the parameters of the entire model so that the learned final feature f_final_i can meet the requirements of small intra-class distance and large inter-class distance. In step 401, the triplet loss is calculated. The classic triplet loss in metric learning is used to train the entire model (from encoder to GNN), that is, in step 402, backpropagation and model optimization are performed. For a randomly selected sample (anchor a), a positive sample p with the same identity and a negative sample n with a different identity are selected. The loss function is defined as follows: L = max( D(f_final_a, f_final_p) - D(f_final_a, f_final_n) + α, 0 ), where: f_final is the final feature vector obtained after the corresponding sample goes through the above complete process. D(x, y) is the distance function between two vectors, such as Euclidean distance. α is a hyperparameter called the margin, which requires that the distance between negative sample pairs be at least α greater than the distance between positive sample pairs. By minimizing this loss function, the model is driven to learn a high-quality feature space. After the model training is completed, when it is necessary to perform person retrieval, in the retrieval phase of this embodiment, in step 501, a feature library of known persons is constructed. In step 502, the features of the query person are extracted. The video clips of all known persons in the database are processed, and their final features f_final are extracted and stored to form a feature library. For a person to be queried, this embodiment also extracts the query person features f_final_q, and in step 503, the distance is calculated and sorted. The distance between it and all features in the library is calculated, and the one with the closest distance is the final matching result.

[0169] As an example, see Figure 16 , Figure 16 This is a schematic diagram of the object graph structure provided in the embodiments of this application. Figure 2 , Figure 16 The object structure diagram shown includes nodes 21, 22, 23, 24 and 25. There is an interaction relationship between nodes 21 and 22, between nodes 21 and 23, between nodes 22 and 24, between nodes 23 and 24, and between nodes 24 and 25.

[0170] This solves the problem of misidentification when people change clothes, greatly enhancing the stability of identity recognition. Traditional methods rely solely on appearance, leading to confusion when someone changes clothes. This application combines a person's appearance, name, and voice. Clothes may change, but a character's name and voice remain largely constant within a work, creating a more reliable and long-term identity profile for each person, ensuring continuity of identity across long periods or multiple scenarios. It also solves the problem of face blindness when distinguishing similar individuals, significantly improving recognition accuracy. Traditional methods fail when multiple people are wearing uniforms or have similar appearances. This application analyzes relationships between individuals, examining who they frequently appear with. Utilizing this contextual information, even if two people look almost identical, their social circles can be used to accurately separate them, avoiding numerous recognition errors. This fundamentally improves the accuracy and intelligence of recognition, making machine recognition more human-like. While related technologies rely on image guessing, this application employs comprehensive reasoning. By utilizing previously wasted text and relationship information in videos, this application provides machines with multiple, mutually verifying clues for judgment. This makes the final recognition result no longer a fragile guess based on a single vision, but a reliable conclusion supported by multi-dimensional information, with overall accuracy and robustness far exceeding traditional methods.

[0171] It is understood that in this application embodiment, data such as video is involved. When this application embodiment is applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0172] The following description continues to illustrate the exemplary structure of the identity recognition device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the identity recognition device 455 in the memory 450 may include: a feature extraction module 4551, used to extract features from each object in the video to obtain a first object feature for each object; a construction module 4552, used to construct an object graph structure of the video using the first object feature as nodes and the interaction relationships between different objects in the video as edges; an update module 4553, used to update the first object feature of each object based on the object graph structure to obtain a second object feature for each object; and an identity recognition module 4554, used to perform identity recognition on each object based on the second object feature of the object to obtain the object identity of the object.

[0173] In some embodiments, the feature extraction module 4551 is further configured to extract features from the audio in the video to obtain a first audio feature of the video, and extract features from the subtitles in the video to obtain a first text feature of the video; perform object recognition on the video to obtain a plurality of objects in the video, and perform the following processing for each object: determine a target video frame including the object from the video frames of the video, and determine the image features of the object based on the target video frame; determine a first object feature of the object based on the image features, the first audio feature, and the first text feature.

[0174] In some embodiments, the feature extraction module 4551 is further configured to, in response to the number of target video frames being one, extract features from the sub-video frames of the target video frame that include the object to obtain image features of the object; in response to the number of target video frames being multiple, extract features from the sub-video frames of each target video frame that include the object to obtain sub-video frame features corresponding to each target video frame; and fuse the sub-video frame features corresponding to each target video frame to obtain image features of the object.

[0175] In some embodiments, the feature extraction module 4551 is further configured to determine a second audio feature that matches the image feature from the first audio feature, and to determine a second text feature that matches the image feature from the first text feature; and to fuse the image feature, the second audio feature and the second text feature to obtain a first object feature of the object.

[0176] In some embodiments, the first audio feature includes first sub-audio features of multiple playback moments of the video, and the image feature includes sub-video frame features corresponding to the target video frame; the feature extraction module 4551 is further configured to, for each first sub-audio feature in the first audio feature, compare the playback moment corresponding to the first sub-audio feature with the playback moment corresponding to the sub-video frame feature to obtain a comparison result corresponding to the first sub-audio feature; in response to the comparison result indicating that the playback moment corresponding to the first sub-audio feature is the same as the playback moment corresponding to the sub-video frame feature, determine the first sub-audio feature corresponding to the comparison result as a second sub-audio feature; and based on the second sub-audio feature, determine a second audio feature that matches the image feature.

[0177] In some embodiments, the first text feature includes first sub-text features of multiple playback moments of the video, and the image feature includes sub-video frame features corresponding to the target video frame; the feature extraction module 4551 is further configured to, for each first sub-text feature in the first text feature, compare the playback moment corresponding to the first sub-text feature with the playback moment corresponding to the sub-video frame feature to obtain a comparison result corresponding to the first sub-text feature; in response to the comparison result indicating that the playback moment corresponding to the first sub-text feature is the same as the playback moment corresponding to the sub-video frame feature, determine the first sub-text feature corresponding to the comparison result as a second sub-text feature; and determine a second text feature matching the image feature based on the second sub-text feature.

[0178] In some embodiments, the identity recognition device 455 further includes: a determination module, configured to perform the following processing for each of the objects in the video: determining the object as a first object, determining objects in the video that are different from the first object as second objects; and determining, based on video frames and subtitles in the video, a third object that has the interaction relationship with the first object from among a plurality of second objects.

[0179] In some embodiments, the determining module is further configured to perform the following processing for each of the second objects: based on the first object and the second object, identify video frames in the video to obtain identification results; in response to the identification results indicating that the video frames include the first object and the second object, retrieve subtitles in the video to obtain retrieval results; in response to the retrieval results indicating that the subtitles contain dialogue between the first object and the second object, determine the second object as a third object that has an interactive relationship with the first object.

[0180] In some embodiments, the determining module is further configured to, for each video frame in the video, determine the video frame as a first video frame when the number of objects present in the video frame is one, and determine the video frame as a second video frame when the number of objects present in the video frame is multiple, and the multiple objects present in the video frame include the first object and the second object; in response to all video frames in the video being the first video frame, determine the identification result as no video frame including the first object and the second object; in response to the presence of the second video frame in the video frame, determine the identification result as a video frame including the first object and the second object.

[0181] In some embodiments, the update module 4553 is further configured to perform the following processing for each of the objects: determine the target object that has an interactive relationship with the object from the object graph structure; call the update model to update the first object feature of the object based on the first object feature of the target object to obtain the second object feature of the object.

[0182] In some embodiments, the update model includes an i-th update layer. The aforementioned update module 4553 is further configured to call the first update layer to update the first object feature of the target object based on the first object feature of the target object, thereby obtaining the first intermediate feature of the object; iteration i performs the following processing: calling the i-th update layer to update the (i-1)-th intermediate feature of the target object based on the (i-1)-th intermediate feature of the target object, thereby obtaining the i-th intermediate feature of the object; and determining the N-th intermediate feature of the object as the second object feature of the object.

[0183] In some embodiments, the i-th update layer includes a fusion layer and an activation layer. The update module 4553 is further configured to call the fusion layer to fuse the (i-1)-th intermediate feature of the target object and the (i-1)-th intermediate feature of the object to obtain the (i-1)-th fused feature; and call the activation layer to activate the (i-1)-th fused feature to obtain the i-th intermediate feature of the object.

[0184] In some embodiments, the update module 4553 is further configured to obtain an object sample graph structure with the first sample features of the object samples as nodes, and call an initial update model to update each first sample feature in the object sample graph structure based on the object sample graph structure to obtain a second sample feature for each object sample; determine the loss value of the initial update model based on the second sample feature, and train the initial update model based on the loss value to obtain the update model.

[0185] In some embodiments, the identity recognition module 4554 is further configured to obtain a preset mapping relationship between multiple preset object identities and multiple preset object features, wherein the preset mapping relationship includes a preset index entry between each preset object identity and the corresponding preset object feature; query the preset object feature that is closest to the second object feature from the preset mapping relationship, and determine the preset index entry where the preset object feature is located as the target index entry of the object; and determine the preset object identity in the target index entry as the object identity of the object.

[0186] In some embodiments, the identity recognition module 4554 is further configured to invoke an identity recognition model, perform identity recognition on the object based on the object's second object features, obtain the probability of the object corresponding to each preset object identity, and determine the preset object identity with the highest probability as the object identity of the object.

[0187] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the identity recognition method described in this application.

[0188] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the identity recognition method provided in this application. For example, ... Figure 3 The identification method shown.

[0189] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of electronic devices including one or any combination of the above-mentioned memories.

[0190] In some embodiments, computer-executable instructions or computer programs may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0191] As an example, a computer executable instruction or computer program may, but does not necessarily, correspond to a file in a file system. It may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0192] As an example, computer-executable instructions or computer programs may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0193] In summary, the embodiments of this application have the following beneficial effects: (1) By extracting the first object feature for each object in the video, the initial object attribute basis is provided for identity recognition. The object graph structure is constructed with the first object feature as the node and the interaction relationship between different objects as the edge. This can systematically associate the association information of each object in the video, break the isolation of the single object feature, and provide comprehensive association dimension support for subsequent feature updates. Based on the object graph structure, the first object feature of each object is updated to obtain the second object feature. The second object feature not only retains the core attributes of the object itself, but also incorporates the interaction and association attributes with other objects, making up for the limitation of the first object feature containing only its own attributes. This makes the obtained second object feature more distinguishable. By performing identity recognition based on the second object feature, relying on more complete and more distinguishable feature data, the misjudgment caused by insufficient feature dimensions or one-sided information is reduced. By supplementing the key information required for identity recognition, the impact of similar object confusion and incomplete feature information on the recognition results is effectively reduced, thereby effectively improving the accuracy of identity recognition.

[0194] (2) The implementation method of determining object image features based on target video frames can flexibly adopt corresponding processing strategies according to the number of target video frames. When there is only one target video frame, by accurately locating the sub-video frame containing the object and extracting features, the accurate image features of the object at the current moment can be quickly obtained, avoiding interference from background elements on feature extraction and ensuring the effectiveness and specificity of feature data in a single frame scene. When there are multiple target video frames, by extracting features from the sub-video frames of each target video frame and performing fusion processing, the feature information of the object at different times, different angles or different environments can be integrated, correcting possible deviations in single frame features (such as contour deformation caused by shooting angle, light changes leading to...). This approach minimizes color deviations and other artifacts, comprehensively capturing the complete attributes of objects, resulting in more complete and reliable image features. This differentiated processing method can adapt to variations in the number of target video frames in different application scenarios. Whether it's a single target video frame formed by the brief appearance of a shelf in security inspection, multiple target video frames formed by multi-angle shooting of props in film production, or multiple target video frames formed during vehicle movement in intelligent transportation, it can effectively output object image features that meet the needs of the scenario. This provides high-quality data support for subsequent operations such as object identification and object interaction relationship analysis based on image features, reducing subsequent processing errors caused by inaccurate or incomplete features, and improving the efficiency and accuracy of the overall video processing workflow.

[0195] (3) The implementation method of determining the first object feature based on image features, first audio features and first text features is to obtain the second audio features and second text features by filtering information that matches the image features from the first audio features and first text features. This can effectively eliminate audio and text interference data that are irrelevant to the object, and ensure that the included audio and text information is directly related to the object attributes, avoiding feature redundancy or deviation caused by irrelevant information. Then, the image features are fused with the filtered second audio features and second text features, which not only retains the visual attributes of the object (such as appearance, action, position, etc. in the image features), but also supplements the auditory attributes (such as sound content, type, etc. in the second audio features) and text attributes (such as identification, scene annotation, etc. in the second text features) related to the object, so that the first object features cover multi-dimensional information. This multi-feature fusion approach provides a more comprehensive and accurate representation of object attributes. For example, in security detection scenarios, it can combine pedestrian visual features, dialogue audio, and regional text annotations to fully present the pedestrian's status. In film and television production scenarios, it can accurately define prop attributes through armor visual features, collision audio, and prop text annotations. In intelligent transportation scenarios, it can comprehensively reflect bus information based on bus visual features, station announcement audio, and route text annotations. This multi-feature fusion approach can adapt to the feature requirements of objects in different application scenarios, providing richer and more reliable basic data for subsequent construction of object graph structures, updating object features, and conducting identity recognition. It reduces the information loss or misjudgment problems that may exist in single-dimensional features, improves the accuracy and effectiveness of object-related operations in the overall video processing workflow, and reduces invalid data processing and computational resource consumption through feature matching and filtering.

[0196] (4) The implementation method of determining the second audio feature that matches the image feature from the first audio feature is to compare the playback time corresponding to each first sub-audio feature in the first audio feature with the playback time corresponding to the sub-video frame feature in the image feature. Only the first sub-audio features with the same playback time are determined as the second sub-audio features. The second audio feature is determined based on the second sub-audio features. This method can strictly control the correlation between audio features and image features from the time dimension, ensuring that the selected second audio feature is the audio information generated at the same time as the object's presentation state in the target video frame. This effectively eliminates the influence of irrelevant audio at different times (such as ambient sound when the object does not appear in the video frame, or interference sound at other times) on the matching result, and avoids the influence of time misalignment. This addresses the issue of mismatch between audio and image features, improving the accuracy of the association between the second audio feature and the image feature. This playback-time-based matching method eliminates the need for complex feature attribute comparison algorithms, achieving effective filtering solely through consistency judgment in the time dimension. The operation logic is simple and efficient, reducing the computational load in the feature matching process and lowering the performance requirements of processing equipment. At the same time, it ensures the reliability of the filtering results, enabling the first object feature obtained by fusing image features, second audio features, and second text features to have higher consistency and accuracy in the time dimension. This provides multi-dimensional data support for subsequent construction of object graph structures, updating object features, and conducting identity recognition, further reducing subsequent processing errors caused by feature time asynchrony.

[0197] (5) By comparing the playback time of each first sub-text feature with that of the sub-video frame feature, only the first sub-text features with the same time are identified as the second sub-text features and further integrated to form the second text feature. This can strictly lock the text information that is in the same state as the object in the target video frame from the time dimension, effectively eliminating the interference of irrelevant text at different times (such as other area annotations when the object does not appear in the video frame, simple timestamp subtitles) on the matching results, avoiding the problem of mismatch between text and image features due to time misalignment, and significantly improving the accuracy of the association between the second text feature and the image feature. This matching method based on playback time does not require complex text semantics and image content association algorithms. It can complete the effective screening by simply judging the time consistency. The operation logic is simple. It is concise and efficient, reducing the computational load in the feature matching process, lowering the performance requirements of processing equipment, and ensuring the reliability of the screening results. By integrating the second sub-text features of time-matching to form the second text features, it can completely cover the synchronous text information of each corresponding playback time of the object, such as the area and product activity information of the snack shelf in the security detection scene, the scene and material information of the prop table in the film and television production scene, and the road and effective time information of the speed limit sign in the intelligent transportation scene. It provides time-synchronized text data support for the subsequent fusion of image features, second audio features and second text features to obtain accurate first object features, further reducing subsequent processing errors caused by feature time asynchrony, and improving the stability and effectiveness of object-related operations in the overall video processing workflow.

[0198] (6) The process of extracting audio and subtitle features from the video, identifying objects, and determining the target video frame, image features, and first object features for each object. By extracting the first audio features and the first text features of the video respectively, the sound and text information at different times in the video can be fully captured, avoiding the limitations of single visual dimension information and providing rich multi-dimensional data support for subsequent object feature analysis. By identifying multiple objects in the video, the independent entities in the video content can be accurately distinguished, laying the foundation for subsequent personalized processing for each object. For each object, the target video frame containing the object can be determined, eliminating the interference of blank frames without objects, focusing on the effective frame image to extract the image features of the object, and improving the efficiency and accuracy of image feature extraction. Based on the image features, the first audio features, and the first text features, the first object features of the object are determined. By filtering and fusing the audio and text information that match the image features at the same time, the first object features are ensured to include both the visual attributes of the object and the auditory and text attributes of the same period, making the feature information more comprehensive and complete.

[0199] (7) The method of identifying video frames based on the first object and the second object and obtaining the result: By dividing the first video frame and the second video frame according to the number of objects in the video frame and whether it contains two types of objects at the same time, the frame images of the coexistence of the two types of objects can be accurately screened, avoiding misjudging single object frames or multi-object frames containing other objects as frames of the coexistence of the two types of objects, and significantly improving the accuracy of frame classification; the final recognition result is determined based on whether there is a second video frame in all video frames, which can fully cover the video content and avoid recognition deviations caused by missing some frames. For example, in the security detection scenario, it can accurately determine whether pedestrians and shopping carts have been in the same frame, and in the film and television production scenario, it can confirm whether actors and shopping carts have been in the same frame. Whether sword props appear simultaneously, and whether buses and traffic lights coexist in intelligent transportation scenarios; the entire process can be completed simply by detecting the number and type of objects, without the need for complex behavior analysis algorithms. The operation logic is simple and efficient, which can reduce the consumption of computing resources and quickly output clear recognition results. This provides a reliable frame-level judgment basis for further analysis of the interaction relationship between the two types of objects (such as determining whether there is a pushing relationship between pedestrians and shopping carts, or whether there is a holding relationship between actors and sword props), reduces invalid operations in subsequent interaction relationship analysis, improves the efficiency and reliability of the overall object association processing flow, and adapts to the needs of object coexistence recognition in various application scenarios.

[0200] (8) Based on the process of identifying the third object through video frame recognition and subtitle retrieval of the first and second objects, the process first confirms whether the two types of objects are in the same frame through video frame recognition, providing a visual basis for the determination of interaction relationship and avoiding the one-sidedness of the judgment caused by relying on a single information source; then, for the case where there are frames in the same frame, the corresponding time period subtitles are retrieved to verify whether there is a related dialogue between the two types of objects, supplementing the interaction evidence that cannot be directly presented by visual information, such as confirming the relationship between pedestrians and shopping carts through dialogue in security detection scenarios, verifying the relationship between actors and sword props through dialogue in film and television production scenarios, and clarifying the operational relationship between drivers and dispatch terminals through dialogue in intelligent transportation scenarios, effectively avoiding misjudgment caused by objects in the same frame but without actual interaction (such as pedestrians and shopping carts accidentally in the same frame but without contact); finally, the second object is determined as the third object only when both the same frame and dialogue are satisfied, which significantly improves the accuracy of the determination of interaction relationship and provides reliable interaction relationship data support for subsequent operations such as constructing object graph structure and updating object features, reducing the interference of invalid data on subsequent processes.

[0201] (9) The above-mentioned update model, which includes the i-th update layer, generates the second object feature through a hierarchical iterative approach. The first update layer supplements the basic association attributes based on the first object features of the target object and the current object. The i-th update layer in subsequent iterations... The update layer leverages the intermediate features of the previous layer to mine deeper correlation information, avoiding insufficient mining or inaccuracy caused by processing large amounts of correlation data in a single update. This allows object features to be progressively refined with iteration, extending from basic regional and temporal correlations to behavioral and impact correlations, significantly improving the completeness and accuracy of the second object's features. For example, in security detection scenarios, pedestrian features can be gradually supplemented with deeper attributes such as contact pushing and positional changes from the same area as the shopping cart; in film production scenarios, actor features can be gradually supplemented with attributes such as handheld use actions from the same scene as the sword prop. Simultaneously, the algorithm complexity of each update layer can be flexibly adapted to scenario requirements. The value of N can be adjusted according to the depth requirements of the correlation attributes, satisfying both basic correlation update needs in simple scenarios and deep correlation mining needs in complex scenarios, enhancing model adaptability. During iteration, the intermediate features of the target object and the current object are updated synchronously, ensuring that the correlation information of the two corroborates each other, avoiding correlation bias caused by lag in one-sided features, and improving the consistency of feature updates. The final generated Nth... Intermediate features, as the second set of object features, can comprehensively cover the object's own attributes and multi-dimensional related attributes, providing high-quality data support for subsequent operations such as object identification and video semantic analysis. This reduces subsequent processing errors caused by missing features. At the same time, automated hierarchical iteration requires no manual intervention, reducing operating costs and error risks, and further improving the efficiency, reliability, and practicality of the overall video processing workflow, adapting to the needs of various application scenarios.

[0202] (10)The above-mentioned $i$-th update layer realizes the update of the intermediate features of the object through the collaborative processing of the fusion layer and the activation layer. The fusion layer can perform attribute association and information integration on the $i - 1$ intermediate features of the target object and the current object, effectively mining the association relationship between the two types of features (such as the same-region attribute and the concurrent sound attribute association between pedestrians and shopping carts in the security detection scenario), eliminating duplicate and redundant information, and integrating effective associated content to generate a fusion feature containing the organically combined information of both, avoiding the omission of associated information caused by the isolated processing of the two types of features, and providing comprehensive basic data for subsequent feature processing; the activation layer then selects the effective attributes directly related to the object interaction in the fusion feature through effectiveness evaluation (such as the shopping cart friction sound attribute), eliminates irrelevant and invalid attributes (such as the shopping cart manufacturer attribute), and at the same time strengthens the feature intensity of the effective attributes, avoiding the interference of invalid information on subsequent processing, and significantly improving the accuracy and usability of the intermediate features; the collaborative work of the two enables the $i$-th intermediate feature to retain the core attributes of the object and the previous associated information, and at the same time adds the effective associated attributes of the target object without redundant interference. Compared with a single processing link, it can achieve the dual effects of comprehensive integration and precise screening of features, ensuring that the quality of the intermediate features after each layer update gradually improves, providing high-quality data support for subsequent iterative updates (if $i < N$) or generating the second object feature (if $i = N$), reducing the subsequent processing errors caused by insufficient feature quality. At the same time, each layer is processed according to this logic to ensure the stability and reliability of the overall update model, adapting to the needs of object feature updates in various scenarios such as security detection, film and television production, and intelligent transportation, and further improving the efficiency and accuracy of the feature update link in the overall video processing process.

[0203] (11) The training method of obtaining the updated model by acquiring the object sample graph structure, calling the initial update model to generate the second sample features, and optimizing the model based on the loss value is consistent with the object graph structure construction logic in actual applications. It can provide data support for model training that is close to the real scene and avoid insufficient model adaptability due to the disconnect between sample data and actual scene. Calling the initial update model to update the first sample features according to the hierarchical iteration logic to obtain the second sample features simulates the complete process of feature update in actual applications, so that the model can grasp the law of feature association update in the training stage and reduce the adaptation error in actual applications. Based on the second sample features, the loss value is determined (such as using triplet loss to quantify the deviation between the feature and the real association) and the model is optimized. It can refine the model through loss feedback. The model parameters (such as the association weights of the fusion layer and the screening thresholds of the activation layer) are precisely adjusted to ensure that the model gradually acquires the ability to distinguish the relationship between samples and accurately update features, avoiding feature update deviations caused by unreasonable model parameters. The entire training process forms a closed loop from data preparation to model optimization. The updated model obtained by training can stably and accurately update the first object features of objects based on the object graph structure in practical applications, generating complete and accurate second object features, providing high-quality data support for subsequent operations such as object identification and video semantic analysis. At the same time, this training method can be flexibly adapted to various scenarios such as security detection, film and television production, and intelligent transportation. By replacing the object sample graph structure of the corresponding scenario, an updated model adapted to the scenario can be trained, significantly improving the model's versatility and practicality, and further enhancing the reliability and efficiency of the overall technical solution in different scenarios.

[0204] (12) By obtaining the preset mapping relationship, querying the target index entry, and determining the object identity, the preset mapping relationship stores the preset index entries containing the preset object identity and corresponding features according to the scenario classification. This can provide a standardized and scenario-appropriate comparison basis for identity recognition, avoiding recognition confusion caused by the lack of unified standards. For example, in the security detection scenario, the entries classified by personnel and items can accurately adapt to the needs of shopping mall object recognition, and in the film and television production scenario, the entries classified by props and equipment can efficiently support the recognition of film set objects. By querying the preset object feature that is closest to the second object feature from the preset mapping relationship and determining the target index entry, the target index entry can be accurately screened through multi-dimensional feature distance calculation and comprehensive comparison. The system identifies the most matching features and corresponding entries, avoiding misidentification caused by single-dimensional feature comparison. For example, in smart transportation scenarios, it combines vehicle color, route markings, and interactive relationships to calculate distances from multiple dimensions, accurately identifying the bus. It determines the preset object identity in the target index entry as the object identity, directly relying on standardized entries to complete identity mapping without additional complex calculations, significantly improving identity recognition efficiency. The entire process is tightly integrated, ensuring the accuracy and reliability of identity recognition, while also flexibly adapting to various scenarios such as security detection, film and television production, and smart transportation through categorized and stored preset mapping relationships. At the same time, the automated query and matching process reduces manual intervention, lowering operational errors and costs.

[0205] (13) The model calls the identity recognition model to obtain the probability of each preset object identity corresponding to the object based on the second object features, and determines the preset object identity with the highest probability as the object identity. The model can effectively mine the core information related to the identity in the features by standardizing the second object features (such as converting facial contours into feature vectors and normalizing color parameters) and extracting key features (such as facial key points and material texture), avoiding recognition deviations caused by chaotic original feature dimensions or omission of key information; it uses methods such as cosine similarity and Pearson correlation coefficient to calculate the matching degree and obtain the probability by normalization and softmax function processing, so as to intuitively reflect the matching degree between the object and each preset identity in a quantitative form, so that identity recognition is upgraded from qualitative judgment to quantitative analysis, which is more scientific than simple feature comparison. At the same time, the probability distribution can clearly present the matching confidence, reducing the risk of misjudgment; the identity with the highest probability is determined as the object identity, without complex manual decision-making, which can quickly lock the most likely identity, significantly improve the recognition efficiency, and can be flexibly adapted to different scenarios. In security detection scenarios, probability calculations based on facial and interactive features ensure accurate identification of personnel. In film and television production scenarios, probability calculations based on material and scene interaction features enable accurate identification of props. In intelligent transportation scenarios, probability calculations based on vehicle body and route marking features complete vehicle identification. The entire process is highly automated, reducing errors caused by human intervention. At the same time, the probabilistic results provide quantitative basis for subsequent possible identity verification, further enhancing the reliability and practicality of the identity recognition process. This lays a precise foundation for subsequent operations such as behavior detection and scene management based on clearly identified objects, improving the efficiency and adaptability of the overall video processing workflow.

[0206] (14) By identifying target objects with interactive relationships from the object graph structure and calling the update model to update the first object features of the target objects based on the first object features of the target objects, the contextual information associated with the current object can be effectively integrated into the feature representation of the object. This makes the second object features of the object not only contain its own local information, but also integrate the global association information brought about by the interaction relationship, thereby significantly enhancing the discriminative ability and richness of the features. This graph-based relational modeling and feature update mechanism can effectively capture the semantic dependencies, temporal relationships or spatial connections between objects, and overcome the feature ambiguity problem caused by incomplete information and missing context when relying only on the local features of a single object. This makes the updated second object features more accurately reflect the true attributes and states of the objects. Applying this second object feature rich in association information to the speech signal classification task can provide a more comprehensive and robust decision basis for the classification model, reduce misclassification caused by one-sided or isolated feature information, thereby effectively improving the classification accuracy of speech signals and ensuring stable and excellent classification performance in complex and variable acoustic environments.

[0207] (15) When processing each object in the video separately, the current object is first distinguished from other objects in the video. This can avoid interference in the subsequent judgment of interaction relationship caused by the mixing of information between different objects, and lay a clear object distinction foundation for the accurate selection of interactive objects. Then, based on the visual information contained in the video frame (such as the action coordination between objects, the proximity relationship between positions, the orientation association, etc.) and the text information contained in the video subtitles (such as the semantic content describing the interaction behavior between objects, the dialogue association expression, etc.), the objects that have an interaction relationship with the current object are determined from other objects. This can combine the complementary advantages of the two types of information, avoid the misjudgment or omission of interaction relationship caused by relying on only a single information source (such as relying solely on visual information, which is prone to missing implicit interactions without obvious actions, and relying solely on text, which is prone to misjudging semantically ambiguous association expressions), and ensure that the association between the selected interactive objects and the current object is real and accurate. This precisely defined interaction relationship provides correct association information to support subsequent updates of the current object's features based on the characteristics of the interacting object. It avoids object feature bias introduced by erroneous association information, making the updated object features more accurately reflect the object's true attributes and associated context. This provides a more reliable feature basis for speech signal classification, reduces classification bias caused by inaccurate object association information, and ultimately effectively improves the classification accuracy of speech signals.

[0208] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An identity recognition method, characterized by, The method comprises: performing feature extraction on each object in a video to obtain first object features of each object; constructing an object graph structure of the video by taking the first object features as nodes and interaction relationships between different objects in the video as edges; updating the first object features of each object based on the object graph structure to obtain second object features of each object; performing identity recognition on each object based on the second object features of the object to obtain an object identity of the object.

2. The method of claim 1, wherein, The performing feature extraction on each object in the video to obtain first object features of each object comprises: performing feature extraction on audio in the video to obtain first audio features of the video, and performing feature extraction on subtitles in the video to obtain first text features of the video; performing object recognition on the video to obtain a plurality of objects in the video, and performing the following processing on each object respectively: determining target video frames including the object from video frames of the video, and determining image features of the object based on the target video frames; determining first object features of the object based on the image features, the first audio features and the first text features.

3. The method of claim 2, wherein, The determining image features of the object based on the target video frames comprises: in response to the number of the target video frames being one, performing feature extraction on sub-video frames including the object in the target video frame to obtain the image features of the object; in response to the number of the target video frames being a plurality, performing feature extraction on sub-video frames including the object in each of the target video frames to obtain sub-video frame features corresponding to each of the target video frames; fusing the sub-video frame features corresponding to each of the target video frames to obtain the image features of the object.

4. The method of claim 2, wherein, The determining first object features of the object based on the image features, the first audio features and the first text features comprises: determining second audio features matching the image features from the first audio features, and determining second text features matching the image features from the first text features; fusing the image features, the second audio features and the second text features to obtain the first object features of the object.

5. The method of claim 4, wherein, The first audio features comprise first sub-audio features of a plurality of playing moments of the video, and the image features comprise sub-video frame features corresponding to the target video frames; The determining second audio features matching the image features from the first audio features comprises: for each first sub-audio feature in the first audio features, comparing a playing moment corresponding to the first sub-audio feature with a playing moment corresponding to the sub-video frame features to obtain a comparison result corresponding to the first sub-audio feature; in response to the comparison result indicating that the playing moment corresponding to the first sub-audio feature is the same as the playing moment corresponding to the sub-video frame features, determining the first sub-audio feature corresponding to the comparison result as the second audio feature; determine a second audio feature matching the image feature based on the second sub-audio feature.

6. The method of claim 4, wherein, The first text feature includes a first sub-text feature of a plurality of play times of the video, and the image feature includes a sub-video frame feature corresponding to the target video frame. The determining, from the first text feature, a second text feature matching the image feature includes: For each first sub-text feature in the first text feature, comparing a play time corresponding to the first sub-text feature with a play time corresponding to the sub-video frame feature to obtain a comparison result corresponding to the first sub-text feature. In response to the comparison result indicating that the play time corresponding to the first sub-text feature is the same as the play time corresponding to the sub-video frame feature, determining the first sub-text feature corresponding to the comparison result as a second sub-text feature. The determining, from the first text feature, a second text feature matching the image feature based on the second sub-text feature.

7. The method of claim 1, wherein, Before the constructing the object graph structure of the video with the first object feature as a node and an interaction relationship between different objects in the video as an edge, the method further includes: For each object in the video, the following processing is performed respectively: determining the object as a first object, and determining an object different from the first object in the video as a second object; determining, from a plurality of second objects, a third object having the interaction relationship with the first object based on video frames in the video and subtitles in the video.

8. The method of claim 7, wherein, The determining, from a plurality of second objects, a third object having the interaction relationship with the first object based on video frames in the video and subtitles in the video includes: For each second object, the following processing is performed respectively: identifying video frames in the video based on the first object and the second object to obtain an identification result; in response to the identification result indicating that the first object and the second object are included in the video frames, searching subtitles in the video to obtain a search result; in response to the search result indicating that the first object and the second object have a dialogue in the subtitles, determining the second object as a third object having the interaction relationship with the first object.

9. The method of claim 8, wherein, The identifying video frames in the video based on the first object and the second object to obtain an identification result includes: For each video frame in the video, when the number of objects existing in the video frame is one, determining the video frame as a first video frame, and when the number of objects existing in the video frame is a plurality and the plurality of objects existing in the video frame include the first object and the second object, determining the video frame as a second video frame; in response to all video frames in the video being the first video frames, determining the identification result as not including the first object and the second object in the video frames; In response to the second video frame existing in the video frames in the video, the identification result is determined as the first object and the second object existing in the video frame.

10. The method of claim 1, wherein, The updating is implemented by an updating model, and the first object feature of each object is updated based on the object graph structure to obtain a second object feature of each object, including: The following processing is performed for each object respectively: From the object graph structure, a target object having an interaction relationship with the object is determined; The updating model is called to update the first object feature of the object based on the first object feature of the target object to obtain a second object feature of the object.

11. The method of claim 10, wherein, The updating model comprises an ith updating layer, ; The calling of the updating model, the updating of the first object feature of the object based on the first object feature of the target object, and the obtaining of the second object feature of the object include: A first updating layer is called to update the first object feature of the object based on the first object feature of the target object to obtain a first intermediate feature of the object; The following processing is performed in iteration i: an i-th updating layer is called to update an (i-1)-th intermediate feature of the object based on the (i-1)-th intermediate feature of the target object to obtain an i-th intermediate feature of the object; The N-th intermediate feature of the object is determined as the second object feature of the object.

12. The method of claim 11, wherein, The i-th updating layer includes a fusion layer and an activation layer, and the calling of the i-th updating layer, the updating of the (i-1)-th intermediate feature of the object based on the (i-1)-th intermediate feature of the target object, and the obtaining of the i-th intermediate feature of the object include: The fusion layer is called to fuse the (i-1)-th intermediate feature of the target object and the (i-1)-th intermediate feature of the object to obtain an (i-1)-th fusion feature; The activation layer is called to perform feature activation on the (i-1)-th fusion feature to obtain the i-th intermediate feature of the object.

13. The method of claim 10, wherein, Before the updating of the first object feature of each object based on the object graph structure to obtain the second object feature of each object, the method further includes: An object sample graph structure with first sample features of object samples as nodes is obtained, and an initial updating model is called to update each first sample feature in the object sample graph structure based on the object sample graph structure to obtain a second sample feature of each object sample; A loss value of the initial updating model is determined based on the second sample feature, and the initial updating model is trained based on the loss value to obtain the updating model.

14. The method of claim 1, wherein, The identity recognition of the object based on the second object feature of the object to obtain an object identity of the object includes: A preset mapping relationship between a plurality of preset object identities and a plurality of preset object features is obtained, and the preset mapping relationship includes a preset index entry between each preset object identity and a corresponding preset object feature; querying, from the preset mapping relationship, a preset object feature closest to the second object feature of the object, and determining a preset index entry in which the preset object feature is located as a target index entry of the object; determining a preset object identity in the target index entry as an object identity of the object.

15. The method of claim 1, wherein, The identity recognition of the object based on the second object feature of the object includes: calling an identity recognition model to perform identity recognition on the object based on the second object feature of the object, to obtain a probability of each preset object identity corresponding to the object; determining a preset object identity with the maximum probability as the object identity of the object.

16. An identity recognition apparatus characterized by comprising: The apparatus includes: a feature extraction module configured to perform feature extraction on each object in a video to obtain a first object feature of each object; a construction module configured to construct an object graph structure of the video by taking the first object feature as a node and taking an interaction relationship between different objects in the video as an edge; an updating module configured to update the first object feature of each object based on the object graph structure to obtain a second object feature of each object; an identity recognition module configured to perform identity recognition on each object based on the second object feature of the object to obtain an object identity of the object.

17. An electronic device, comprising: The electronic device includes: a memory configured to store computer executable instructions or computer programs; a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the identity recognition method in any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program comprise the steps of: The computer executable instructions or computer programs are executed by the processor to implement the identity recognition method in any one of claims 1 to 15.

19. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the identity recognition method in any one of claims 1 to 15.

Images