Visual data management method and device in vehicle sentry mode, medium and product

Abstract

The invention provides a visual data management method and device in a vehicle sentry mode, a medium and a product, and relates to the technical field of vehicle data management. Performing feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data; the self-supervised feature extraction model is obtained by performing self-supervised learning training on unlabeled sentinel visual sample data based on a general visual feature extraction model; performing hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clusters of the visual data; and based on the implicit identifiers of the plurality of clusters, establishing a semantic index database to perform life cycle management on the visual data. According to the invention, the management efficiency of the visual data generated in the vehicle sentry mode can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle data management, and in particular to a visual data management method, device, medium and product for a vehicle sentry mode. BACKGROUND

[0002] With the rapid popularization of vehicle intelligent networking technology, the vehicle sentry mode has gradually become a standard configuration of many vehicle models from an optional function, and has become one of the core technologies for vehicle parking safety protection.

[0003] However, while the vehicle sentry mode continuously monitors the environment around the vehicle and records potential security threats such as scratches, collisions, and attempted thefts in real time, it also generates a large amount of image and video visual data. In related technologies, the processing of the large amount of visual data generated by the vehicle sentry mode mainly relies on solutions of manually adding labels or designing rule models. However, due to the dependence on manual intervention, the efficiency of managing these visual data is very low. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a visual data management method, device, medium and product for a vehicle sentry mode, aiming to improve the management efficiency of visual data generated by the vehicle sentry mode.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application proposes a visual data management method for a vehicle sentry mode, the method comprising: acquiring visual data in a vehicle sentry mode; performing feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data; the self-supervised feature extraction model is obtained based on a general visual feature extraction model through self-supervised learning training on unlabeled sentry visual sample data; performing hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data; based on the implicit identification of each of the plurality of clustering clusters, establishing a semantic index database to manage the life cycle of the visual data.

[0006] In some embodiments, the target feature vector includes a multi-dimensional feature vector of each frame of the visual data. The hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data comprises: performing frame-level clustering processing on the visual data based on the multi-dimensional feature vector to obtain a static visual clustering cluster of the visual data; the plurality of clustering clusters of the visual data includes the static visual clustering cluster; segment fusion processing is performed on the multi-dimensional feature vectors based on at least one time segment of the visual data, to obtain a frame feature vector fusion result of the at least one time segment; segment-level clustering processing is performed on the visual data based on the frame feature vector fusion result, to obtain a dynamic event clustering cluster of the visual data; and the multiple clustering clusters of the visual data include the dynamic event clustering cluster.

[0007] In some embodiments, the frame-level clustering processing performed on the visual data based on the multi-dimensional feature vectors includes: similarity measurement processing is performed on the multi-dimensional feature vectors of each frame of the visual data, to obtain a similarity measurement result; unsupervised clustering processing is performed on the multi-dimensional feature vectors based on the similarity measurement result, to obtain a cluster division result of the multi-dimensional feature vectors; and the cluster division result includes at least one feature clustering cluster; cluster matching processing is performed on each frame of the visual data based on the at least one feature clustering cluster, to obtain at least one static visual clustering cluster of the visual data.

[0008] In some embodiments, the segment-level clustering processing performed on the visual data based on the frame feature vector fusion result includes: clustering analysis is performed on the frame feature vector fusion result, to obtain cluster information of each frame within each time segment of the visual data; aggregation is performed on the cluster information based on a time sequence relationship of each time segment of the visual data, to obtain a dynamic event clustering cluster of the visual data.

[0009] In some embodiments, the implicit identifier includes a machine semantic label, and the method further includes: aggregation calculation is performed on a target feature vector in a target clustering cluster based on an aggregation function, to obtain a core feature vector of the target clustering cluster; and the target clustering cluster is any one of the multiple clustering clusters; semantic translation is performed on the core feature vector based on a preset visual language large model, to obtain a machine semantic label of the target clustering cluster.

[0010] In some embodiments, the life cycle management performed on the visual data based on the respective implicit identifiers of the multiple clustering clusters includes: an association relationship is established between the respective implicit identifiers of the multiple clustering clusters and metadata of the visual data, to obtain a semantic index database; and the metadata includes segment identifier data of the visual data, a mapping relationship between the visual data and the multiple clustering clusters, and original storage location data of the visual data; store the visual data based on the semantic index database, and perform semantic retrieval on the stored visual data based on the semantic index database.

[0011] In some embodiments, the semantic retrieval on the stored visual data based on the semantic index database comprises: performing semantic similarity calculation on the query intent information corresponding to the visual data query statement and the stored visual data in the semantic index database to obtain a semantic similarity calculation result. In a case where the semantic similarity calculation result indicates that the semantic similarity between the query intent information and target visual data is greater than or equal to a preset threshold, outputting the target visual data as the query result of the visual data query statement; the target visual data is at least one segment of the stored visual data in the semantic index database.

[0012] In some embodiments, the feature extraction processing on the visual data based on the self-supervised feature extraction model to obtain the target feature vector of the visual data comprises: performing key frame sampling on the target segment of the visual data to obtain a key frame sequence; the target segment is at least one of the multiple segments of the visual data; performing feature extraction processing on at least one key frame in the key frame sequence based on the self-supervised feature extraction model, and outputting a multi-dimensional feature vector of the target segment by fusing the extracted multi-dimensional feature vectors of the key frames based on the self-supervised feature extraction model; the target feature vector of the visual data comprises the multi-dimensional feature vector of the target segment.

[0013] To achieve the above object, a second aspect of the embodiment of the present application provides a visual data management device for vehicle sentry mode, which comprises: an acquisition module configured to acquire visual data in vehicle sentry mode; a self-supervised feature extraction module configured to perform feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data; the self-supervised feature extraction model is obtained by self-supervised learning and training of a general visual feature extraction model on unannotated sentry visual sample data; a hierarchical clustering module configured to perform hierarchical clustering processing on the visual data based on the target feature vector to obtain multiple clustering clusters of the visual data; a semantic index module configured to establish a semantic index database based on the implicit identifiers of the multiple clustering clusters to perform life cycle management on the visual data.

[0014] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the vehicle sentry mode visual data management method of the first aspect when executing the computer program.

[0015] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the vehicle sentry mode visual data management method of the first aspect when executed by a processor.

[0016] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer program product, comprising a computer program, which implements the vehicle sentry mode visual data management method of the first aspect when executed by a processor.

[0017] The vehicle sentry mode visual data management method, device, electronic device, computer readable storage medium and computer program product provided by the embodiments of the present application acquire visual data in a vehicle sentry mode, perform feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data, the self-supervised feature extraction model is obtained based on a general visual feature extraction model through self-supervised learning training on unlabeled sentry visual sample data, perform hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data, and establish a semantic index database based on the implicit identifiers of the plurality of clustering clusters to manage the life cycle of the visual data.

[0018] Compared with the traditional method of relying on manual intervention for sentry visual data management, the embodiments of the present application pre-train a general visual feature extraction model based on large-scale unlabeled sentry visual sample data through self-supervised learning training to obtain a self-supervised feature extraction model that can extract general visual features based on self-supervised learning, so that after acquiring visual data generated in a vehicle sentry mode, the self-supervised feature extraction model is used to perform feature extraction processing on the visual data to obtain a target feature vector of the visual data, then hierarchical clustering processing is performed on the visual data based on the target feature vector to obtain a plurality of clustering clusters, and finally a semantic index database is established based on the implicit identifiers of the plurality of clustering clusters to manage the life cycle of the visual data.

[0019] Therefore, in the embodiment of the present application, since the self-supervised learning can be completely free of human intervention and only the massive sentinel visual sample data can be used for training, the participation of manual annotation work can be completely avoided, and the artificial cost and time cost for feature extraction of visual data can be greatly reduced, the efficiency bottleneck caused by dependence on human is solved, and the efficiency of processing visual data is improved. Meanwhile, since the self-supervised learning process of the self-supervised feature extraction model is objective and data-driven, the subjective bias problem caused by human intervention can also be avoided, so that the consistency and reliability of the target feature vector extracted from the visual data are guaranteed.

[0020] In addition, the processing of hierarchical clustering of visual data based on the target feature vector extracted from the visual data in the embodiment of the present application can automatically classify the massive visual data without human intervention to define the category of the visual data, so as to improve the automation level and efficiency of the overall analysis process. Further, the life cycle management of the visual data based on the semantic index database established by the implicit identification of the plurality of clustering clusters can store the data of similar events in the visual data in a centralized manner, which not only facilitates management, but also greatly optimizes the subsequent data utilization efficiency and the retrieval and mining speed of valuable data. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The step flowchart of the vehicle sentinel mode visual data management method provided by the embodiment of the present application in some embodiments is shown in the figure; Figure 2 The detailed step flowchart of step S102 in the embodiment is shown in the figure; Figure 1 Figure 3 The detailed step flowchart of step S103 in the embodiment is shown in the figure; Figure 1 The detailed step flowchart of step S103 in the embodiment is shown in the figure; Figure 4 The step flowchart of the vehicle sentinel mode visual data management method provided by the embodiment of the present application in some other embodiments is shown in the figure; Figure 5 The detailed step flowchart of step S104 in the embodiment is shown in the figure; Figure 1 The detailed step flowchart of step S502 in the embodiment is shown in the figure; Figure 6 Figure 5 The detailed step flowchart of step S502 in the embodiment is shown in the figure; Figure 7 The system overall flowchart of the vehicle sentinel mode visual data management method provided by the embodiment of the present application in a complete embodiment is shown in the figure; Figure 8 The structure diagram of the vehicle sentinel mode visual data management device provided by the embodiment of the present application is shown in the figure; Figure 9 ​​A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In order to make the purposes, technical solutions and advantages of the present application clearer and more apparent, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0023] It should be noted that, although the functional modules are divided in the device / system schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device / system or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0025] First, the overall concept of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application is described.

[0026] With the rapid popularization of vehicle intelligent networking technology, the vehicle sentry mode has gradually become a standard configuration of many vehicle models from an optional function, becoming one of the core technologies for vehicle parking safety protection. The vehicle sentry mode continuously monitors the environment around the vehicle, records potential security threats such as scratches, collisions, and attempted thefts on the vehicle in real time, and also generates a large amount of image and video visual data.

[0027] The visual data generated under the vehicle sentry mode has a very high value for improving vehicle safety, optimizing automatic driving algorithms, and identifying accident liability. However, how to efficiently store, manage and retrieve these unstructured mass visual data has become a huge technical challenge. Traditional solutions mainly rely on manual tagging or simple classification of data based on manually designed rule models, which have inherent defects such as coarse granularity, poor flexibility, low efficiency, and strong subjectivity, thereby causing the great value of visual data under the vehicle sentry mode to be severely restricted by inefficient and rough data management methods. In order to overcome the problems caused by traditional solutions, there are the following several interrelated difficulties: How to meet the precise needs of complex scenarios.

[0028] Traditional solutions can only do "binary classification" (event or no event) or very limited coarse-grained classification (e.g. "person", "vehicle"). However, the real-world application scenarios of vehicle sentry mode are much more complex. For example, just knowing "someone is approaching" is far from enough, it needs to further distinguish "the car owner is taking the car", "a passer-by is passing by", "a courier is placing a package", or "a thief is trying to open the door". Similarly, for the events around the vehicle, it also needs to distinguish "a slight scratch", "a serious collision", or "a neighboring vehicle door impact". Since this kind of fine-grained, scenario-based recognition ability is crucial for accurate alarm, insurance damage assessment, and user experience improvement, and the traditional data management method cannot provide such rich information dimensions, resulting in a large amount of valuable details being submerged. Therefore, how to realize the classification of visual data from "event or no event" to "what event, what object, what cause", and multi-dimensional, deep semantic understanding and classification of events, is the core technical challenge in visual data management.

[0029] How to adapt to unknown or evolving scenarios.

[0030] In traditional solutions, the system based on pre-defined rules is essentially closed, and such a system can only recognize the cases that the programmer has thought of and manually written rules for. For novel attack methods, rare traffic accident forms, or new objects not included in the classification system, the system will be completely "blind". The real world is infinite and constantly changing. This means that traditional systems will quickly become obsolete over time, and maintaining and updating the rule library requires continuous investment with little effect. Therefore, a more advanced technical problem of managing visual data is how to enable the system to have the ability to "discover" and "learn", to automatically mine unknown, potential, and meaningful new patterns or abnormal events from data, and continuously evolve and expand its cognitive boundaries without the need for expensive and time-consuming reprogramming each time.

[0031] How to avoid the credibility of data being compromised.

[0032] The traditional solution cannot guarantee the quality of the output even if a large amount of manpower is invested in data labeling. This is because different labeling personnel may have different judgments on the same scene (for example, the definition of "suspicious behavior"), and even the same labeling personnel may have different standards in a fatigue state and a non-fatigue state. This subjectivity introduces "noise", which makes the accuracy and reliability of the conclusions based on these data questionable. For the training of autonomous driving algorithms, low-quality or inconsistent labels can even pose a safety hazard. Therefore, a deeper problem of visual data management is how to eliminate human subjective factors and ensure the objectivity, consistency and repeatability of data interpretation, so as to ensure the accuracy and fairness of downstream applications (such as model training and responsibility identification).

[0033] How to balance the sharp contradiction between the huge data size and the low processing efficiency.

[0034] The traditional method relying on manual visual inspection or simple rule-based filtering has a gap of several orders of magnitude between the processing speed and the data generation speed. The cost of manual labeling is extremely high, and it is completely impossible to meet the needs of real-time or near real-time processing. This results in a large amount of data being "data garbage" because it cannot be processed in time, and it is either deleted or stored in the hard disk, and cannot play its due value. Therefore, the primary problem of visual data management is how to achieve a qualitative leap in processing speed, so that the data analysis efficiency can match the data generation rate, and the per capita data processing cost can be greatly reduced.

[0035] How to break the fundamental gap between unstructured data and management and retrieval needs Raw visual data such as video and image are unstructured, and they cannot be efficiently indexed and queried by a database. For example, users cannot perform natural language-based retrieval such as "all events involving electric scooters scratching the left side of the door on x year x month x day". To achieve such retrieval, the unstructured data must be converted into structured label text through manual means, and this process is the root of all the above problems. Therefore, the most fundamental technical problem of visual data management is how to break this gap, automatically generate rich, accurate, and machine-readable structured semantic descriptions (metadata) for each piece of unstructured visual data through advanced artificial intelligence technology, and thus lay the foundation for millisecond-level, multi-condition, and intelligent management of massive data.

[0036] To this end, the embodiments of the present application provide a vehicle sentinel mode visual data management method and device, electronic equipment, computer readable storage medium and computer program product, which are aimed at overcoming the shortcomings of the above related technologies and improving the efficiency of processing visual data generated by the vehicle sentinel mode.

[0037] The vehicle sentry mode visual data management method, device, electronic equipment, computer readable storage medium and computer program product provided by the embodiments of the present application, by obtaining visual data in the vehicle sentry mode, performing feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data, the self-supervised feature extraction model is obtained based on self-supervised learning training of a general visual feature extraction model on unlabeled sentry visual sample data, performing hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data, and establishing a semantic index database based on the implicit identifiers of the plurality of clustering clusters to manage the life cycle of the visual data.

[0038] Compared with the traditional way of relying on manual intervention for sentry visual data management, the embodiments of the present application pre-train a general visual feature extraction model based on a large amount of unlabeled sentry visual sample data for self-supervised learning, so as to obtain a self-supervised feature extraction model that can extract general visual features based on self-supervised learning, so that after obtaining the visual data generated in the vehicle sentry mode, the self-supervised feature extraction model is used to perform feature extraction processing on the visual data to obtain a target feature vector of the visual data, then the target feature vector is used to perform hierarchical clustering processing on the visual data to obtain a plurality of clustering clusters, and finally a semantic index database is established based on the implicit identifiers of the plurality of clustering clusters to manage the life cycle of the visual data.

[0039] In this way, in the embodiments of the present application, since self-supervised learning can completely avoid human intervention and only use a large amount of sentry visual sample data for training, the participation of manual annotation work can be completely avoided, thereby greatly reducing the human cost and time cost of feature extraction of visual data, solving the efficiency bottleneck caused by relying on manual work, and improving the efficiency of processing visual data. At the same time, since the self-supervised learning process of the self-supervised feature extraction model is objective and data-driven, the subjective bias problem caused by human intervention can also be avoided, thereby ensuring the consistency and reliability of the target feature vector extracted from the visual data.

[0040] In addition, the hierarchical clustering processing of the visual data based on the target feature vector extracted from the visual data in the embodiments of the present application can automatically classify a large amount of visual data without human intervention to define the categories of the visual data, thereby improving the automation level and efficiency of the overall analysis process. Further, the establishment of a semantic index database based on the implicit identifiers of the plurality of clustering clusters to manage the life cycle of the visual data can store similar event data in the visual data, which not only facilitates management, but also greatly optimizes the subsequent data utilization efficiency and valuable data retrieval and mining speed.

[0041] Next, the vehicle sentry mode visual data management method, device, electronic device, computer readable storage medium and computer program product provided by the embodiments of the present application are specifically described through the following embodiments, and first, the various detailed embodiments of the vehicle sentry mode visual data management method provided by the embodiments of the present application are described in detail.

[0042] It should be noted that in each of the specific embodiments of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user historical data and user location information, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0043] It should be noted that the vehicle sentry mode visual data management method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a vehicle-mounted terminal device (such as a vehicle-mounted computing platform, etc.) on a vehicle, or a terminal device associated with a vehicle, wherein the association of the terminal device with the vehicle means that the terminal device can communicate data with the vehicle based on a network, for example, the terminal device can be a cloud server device that communicates data with the vehicle. In addition, the terminal can also be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The server end can be a background server terminal device, which can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform. The software can be an application, a computer program, and a storage medium carrying the computer program, etc. that implements the vehicle sentry mode visual data management method. It should be understood that based on different design needs of actual applications, in different feasible embodiments, the terminal, server end and software, etc. that apply the vehicle sentry mode visual data management method provided by the embodiments of the present application can of course also be other forms not listed here, and the vehicle sentry mode visual data management method provided by the embodiments of the present application does not specifically limit this.

[0044] Moreover, embodiments can be practiced in a distributed computing environment where tasks are performed by a remote processing device that is located at a remote computer system, over a network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0045] For the sake of understanding and elaboration, the terminal device applying the visual data management method of the vehicle sentry mode provided by the embodiments of the present application is taken as an example in the following, and each specific embodiment of the present application is described in detail. The implementation of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application in any other form of subject can refer to the implementation process of the visual data management method of the sentry mode elaborated in the following.

[0046] Please refer to Figure 1 , Figure 1 The step flowchart of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application is shown in the following. It should be understood that, although the execution order of some method steps is shown in the following and subsequent other step flowcharts, the visual data management method of the vehicle sentry mode provided by the embodiments of the present application can of course adopt an execution order of method steps different from that shown in the figure. That is, Figure 1 The order of the method steps shown in the figure does not constitute a limitation on the execution logic order of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application, and any reasonable change of the order of the method steps shown in the figure should be included in the protection scope of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application. Figure 1 Figure 1 The order of the method steps shown in the figure does not constitute a limitation on the execution logic order of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application, and any reasonable change of the order of the method steps shown in the figure should be included in the protection scope of the visual data management method of the vehicle sentry mode provided by the embodiments of the present application.

[0047] As Figure 1 shown, in some embodiments, the terminal device applying the visual data management method of the vehicle sentry mode provided by the embodiments of the present application can include the following steps S101 to S104.

[0048] ​Step S101: Obtain visual data in a vehicle sentry mode.

[0049] It should be noted that the vehicle sentry mode can be a vehicle sentry function running on the vehicle terminal device, and the vehicle sentry function can be a passive safety system of the vehicle, which aims to monitor the vehicle and its surrounding environment to ensure the safety of the vehicle. Among them, the vehicle sentry function can monitor the surrounding environment when the vehicle is unattended to discover abnormal situations in time, and once a potential threat (such as kicking the car, scratching the car, and violently hitting the car body, which threatens the safety of the vehicle) is detected, the system can automatically start video recording to provide evidence for subsequent accountability. In addition, the vehicle sentry function can also control the vehicle to issue an audible and light alarm and push an alarm notification to the owner through the application APP when detecting that the vehicle is threatened.

[0050] In addition, the visual data can be raw and unstructured video data collected by the vehicle sentry mode using visual sensors (such as multiple high-definition cameras deployed around the vehicle) configured on the vehicle to continuously monitor the surrounding environment of the vehicle.

[0051] In the process of the terminal device starting to run in the vehicle sentry mode to protect the safety of the vehicle, the terminal device continuously obtains the visual data collected by the vehicle sentry mode when monitoring the surrounding environment of the vehicle.

[0052] In some embodiments, when the terminal device obtains the visual data collected by the vehicle sentry mode, the terminal device can control the vehicle sentry mode to dynamically adjust the frame rate and resolution of the video data collected by the visual sensor. For example, the terminal device can control the vehicle sentry mode to use the following adjustment logic: in a silent state (no people / objects appear in the alert range of the vehicle in the pre-labeled or designated vehicle surrounding environment), the camera uses the lowest frame rate (such as 2 frames per second), and the resolution also uses the lowest (such as 320*320), when people / objects appear in the alert range, the camera uses a medium frame rate (such as 30 frames per second) and a medium resolution (such as first through a super-sampling method and then reduced to 320*320), and when people / objects appear in the alert range and the vibration signal is triggered, the camera uses a high frame rate (such as 60 frames per second), and the resolution also uses the highest resolution and lasts for 30s.

[0053] Step S102: performing feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data; the self-supervised feature extraction model is obtained based on self-supervised learning training of a general visual feature extraction model on unlabeled sentry visual sample data.

[0054] It should be noted that the terminal device can pre-train the model in a self-supervised learning manner, and perform self-supervised learning training on the general visual feature extraction model based on a super-large unlabeled sentinel visual sample data, so as to obtain a trained self-supervised feature extraction model. Through pre-training processing, the self-supervised feature extraction model can not perform feature extraction for any specific category (such as “scratching” and “theft”) in the training and actual application process, but can autonomously learn the general visual representation of the vehicle surrounding environment through reconstruction of the masked image block or instance comparison, so as to extract a high-density and semantic feature vector from the visual data.

[0055] After obtaining the visual data, the terminal device further inputs the visual data into the pre-trained self-supervised feature extraction model, so as to perform feature extraction processing on the visual data by the self-supervised feature extraction model, and obtain the target feature vector of the visual data. Among them, the target feature vector output by the self-supervised feature extraction model through the feature extraction processing on the visual data can efficiently represent the visual and time sequence information of the video, thereby providing a strong basis for downstream tasks (such as behavior recognition, video retrieval, content analysis, etc.).

[0056] In some embodiments, when the terminal device performs feature extraction processing on the visual data based on the self-supervised feature extraction model, the self-supervised feature extraction model can capture features such as image subtle texture, shape, motion pattern change (such as subtle dents on the car paint, specific actions of the pedestrian's arm) from the visual data, so as to obtain the target feature vector of the visual data. In this way, by capturing the subtle features in the image for subsequent fine-grained classification and recognition, the limitation of traditional solutions that can only perform coarse-grained classification is broken through, so as to meet the precise needs of complex scenes.

[0057] In some embodiments, when the terminal device performs feature extraction processing on the visual data based on the self-supervised feature extraction model, the self-supervised feature extraction model can extract a high-density and semantic feature vector from each frame of image in the visual data, so as to obtain the target feature vector of the whole visual data.

[0058] In other embodiments, the terminal device can also extract a high-density and semantic feature vector from only some key frame images in the visual data based on the self-supervised feature extraction model, and then take the extracted features of the key frames as the target feature vector of the whole visual data.

[0059] Step S103: performing hierarchical clustering processing on the visual data based on the target feature vector, to obtain a plurality of clustering clusters of the visual data.

[0060] After the terminal device extracts the target feature vector of the visual data, the terminal device further performs hierarchical clustering on the visual data based on the target feature vector, thereby obtaining a plurality of cluster groups of the visual data.

[0061] In some embodiments, the terminal device can pre-design a hierarchical clustering pipeline. First, a pre-trained self-supervised feature extraction model is used to extract the target feature vector of the visual data at the frame level. Then, based on the time sequence information, the frame sequence is segmented into short video clips, and the features of all frames in the clip are aggregated to form a time sequence feature representation at the clip level. Finally, the visual data is clustered at the frame and clip levels respectively, and the clustering results at different levels are verified and supplemented with each other, thereby obtaining a plurality of cluster groups of the visual data.

[0062] Step S104: Based on the implicit identifiers of the plurality of cluster groups, a semantic index database is established to manage the life cycle of the visual data.

[0063] It should be noted that the implicit identifier can be a machine-generated semantic label assigned to each cluster group by the terminal device after the hierarchical clustering of the visual data produces a plurality of cluster groups. The semantic label can be obtained by querying an external multi-modal model or a keyword library based on the cluster center feature.

[0064] After the terminal device produces a plurality of cluster groups by performing hierarchical clustering on the visual data based on the target feature vector, and assigns an implicit identifier to each cluster group, the terminal device can further establish a semantic index database of the visual data based on the implicit identifier, thereby storing, retrieving, and managing the life cycle of the visual data.

[0065] In some embodiments, when the terminal device establishes the semantic index database of the visual data based on the implicit identifiers of the plurality of cluster groups, the terminal device can take the cluster group with the implicit identifier (semantic label) and the metadata of all clips of the visual data (including the unique identifier ID of the video clip, the cluster identifier ID to which the clip belongs, the mapping relationship between the clip and the cluster in the cluster group, and the original data storage location pointer such as the storage path of the visual data, the uniform resource locator URL, etc.) as input, establish an association index between the video clip and the cluster group, the semantic label, and the original data storage location pointer, and thereby obtain the semantic index database of the visual data. The semantic index database can store information in the form of records, and each record contains fields such as the unique ID of the video clip, the cluster ID, the semantic label, and the original data storage location pointer. In this way, the terminal device can trace and quickly query the visual data based on the semantic index database.

[0066] In the embodiment of the present application, during the process of protecting the vehicle by starting the operation of the vehicle sentry mode by the terminal device, the visual data collected by the vehicle sentry mode monitoring the surrounding environment of the vehicle is continuously obtained. Then, in order to convert the original, unstructured video data into structured, machine-readable high-density feature vector sequence, the visual data is input into the pre-trained self-supervised feature extraction model by the terminal device, so as to obtain the target feature vector of the visual data by the self-supervised feature extraction model. Then, the terminal device performs hierarchical clustering processing on the visual data based on the target feature vector, and obtains a plurality of clustering clusters of the visual data. Finally, the terminal device gives each clustering cluster an implicit identification, and further establishes a semantic index database of the visual data based on the implicit identification, so as to store, retrieve and manage the life cycle of the visual data.

[0067] Compared with the traditional way of relying on manual intervention for sentry visual data management, in the embodiment of the present application, since self-supervised learning can completely not need manual intervention and can be trained only by using massive sentry visual sample data, the participation of manual annotation work can be completely avoided, and the artificial cost and time cost of feature extraction of visual data are greatly reduced, the efficiency bottleneck caused by relying on manual work is solved, and the efficiency of processing visual data is improved. At the same time, since the self-supervised learning process of the self-supervised feature extraction model is objective and data-driven, the subjective bias problem caused by manual intervention can also be avoided, so as to ensure the consistency and reliability of the target feature vector extracted from the visual data.

[0068] In addition, the embodiment of the present application can automatically classify massive visual data without manually defining the categories of visual data by performing hierarchical clustering processing on the visual data based on the target feature vector extracted from the visual data, thereby improving the automation level and efficiency of the overall analysis process. Further, the life cycle management of the visual data by establishing a semantic index database based on the implicit identification of the plurality of clustering clusters obtained by clustering can store similar event data in the visual data, which not only facilitates management, but also greatly optimizes the subsequent data utilization efficiency and the retrieval and mining speed of valuable data.

[0069] Further, in the embodiments of the present application, by abandoning the traditional way of training the model by relying on a large amount of manually annotated data, a pre-trained model based on self-supervised learning is adopted. As such, since the self-supervised feature extraction model learns the general ability of "how to look at the world" rather than the specific ability of "recognizing specific things", the self-supervised feature extraction model can also extract meaningful features for new event types that are unknown or have not been seen before, thereby providing the possibility of discovering new categories and making the system more scalable. As such, compared with the traditional scheme, the embodiments of the present application enhance the flexibility of the system by pre-training the self-supervised feature extraction model to perform feature extraction processing on the visual data, so that the system can adapt to the visual data related processing in the unknown or evolving scene of the vehicle sentinel mode, i.e., based on the "discovery" and "learning" ability of the model, unknown, potential and meaningful new patterns or abnormal events can be automatically mined from the data, and the cognitive boundary of the model can be continuously evolved and expanded without the need for expensive and time-consuming reprogramming each time.

[0070] Please refer to Figure 2 , Figure 2 for Figure 1 the detailed step flowchart of step S102.

[0071] As shown in Figure 2 , in some embodiments, the above-mentioned "feature extraction processing on the visual data based on the self-supervised feature extraction model" in step S102 can include steps S201 and S202 as shown below.

[0072] Step S201: key frame sampling is performed on a target segment of the visual data to obtain a key frame sequence; the target segment is at least one of the multiple segments of the visual data.

[0073] After obtaining the visual data generated in the vehicle sentinel mode, the terminal device can perform video segmentation and key frame sampling preprocessing on the visual data, and then perform feature extraction on the sampled key frames to obtain a high-density feature vector as the target feature vector of the visual data. Based on this, in the preprocessing stage of the visual data, the terminal device can first perform video segmentation processing on the visual data to obtain multiple segments of the visual data, and then further perform key frame sampling on any target segment in the multiple segments to obtain a key frame sequence for each segment.

[0074] In some embodiments, when the terminal device performs preprocessing on the visual data, it can further perform down-sampling processing on each segment after cutting the visual data into shorter segments, thereby extracting the key frame sequence of each segment.

[0075] Exemplarily, assuming that the visual data acquired by the terminal device is an original video , the terminal device first uniformly divides the original video into non-overlapping (or overlapping) segments : }.

[0076] Then, for each segment , the terminal device can sample K key frames therefrom: }.

[0077] wherein represents the k-th sampled frame in the i-th segment.

[0078] In addition, the sampling strategy can be represented as a function S: } = S( ).

[0079] wherein Strategy can be equal-interval sampling, random sampling, etc.

[0080] Step S202: performing feature extraction processing on at least one key frame in the key frame sequence based on the self-supervised feature extraction model, and fusing the extracted key frame multi-dimensional feature vectors into a multi-dimensional feature vector of the target segment based on the self-supervised feature extraction model to output; the target feature vector of the visual data comprises the multi-dimensional feature vector of the target segment.

[0081] After the terminal device samples key frames from each segment of the visual data to obtain a key frame sequence, it sequentially takes each segment as a target segment and inputs the key frame sequence of the target segment into the self-supervised feature extraction model, so as to perform feature extraction processing on the key frame sequence by the self-supervised feature extraction model, i.e., capture the subtle texture, shape, motion pattern change, etc. features of each image in the key frame sequence, and take the extracted subtle features as the key frame multi-dimensional feature vector of each image. Moreover, the terminal device also fuses the key frame multi-dimensional feature vectors of each image in the key frame sequence to obtain a multi-dimensional feature vector of the target segment after extracting the key frame multi-dimensional feature vector of each image by the self-supervised feature extraction model, and outputs the multi-dimensional feature vector. In this way, the terminal device can obtain the multi-dimensional feature vector of each segment output by the self-supervised feature extraction model, and take these multi-dimensional feature vectors as the target feature vector of the visual data.

[0082] Exemplarily, assuming that the self-supervised feature extraction model is a function wherein is the parameter of model pre-training, the model takes the key frame sequence sampled as input, and performs fine feature extraction for each frame image to obtain the feature of each frame image:

[0083] wherein, is the D-dimensional high-density feature vector (key frame multi-dimensional feature vector) corresponding to the k-th key frame.

[0084] Then, the self-supervised feature extraction model outputs the global feature Z of the entire segment by fusing the features of all key frames in the sequence: Z =

[0085] wherein, is the global D-dimensional feature vector (multi-dimensional feature vector) fused with the feature information of all K frames.

[0086] In some embodiments, the terminal device can fuse the key frame multi-dimensional feature vectors of all frames in the target segment into a multi-dimensional feature vector of the target segment by averaging the key frame multi-dimensional feature vectors. Alternatively, the terminal device can fuse the key frame multi-dimensional feature vectors of all frames in the target segment into a multi-dimensional feature vector of the target segment by using a special token [CLS] in the self-supervised feature extraction model.

[0087] In this embodiment, the terminal device samples key frames from the target segment of the visual data to obtain a key frame sequence, and then performs feature extraction processing on the key frames in the key frame sequence based on the self-supervised feature extraction model. In this way, the self-supervised feature extraction model can learn from the data to extract a multi-dimensional feature vector without any manual annotation, thereby realizing an end-to-end automatic pipeline from feature extraction to event classification. Moreover, by automatically discovering the intrinsic pattern in the data through unsupervised clustering, the system can autonomously induce and discover unknown new event types, thereby enabling the system to have the ability of self-evolution and continuous learning.

[0088] In some embodiments, the target feature vector includes a multi-dimensional feature vector of each frame of the video data.

[0089] It should be noted that the multi-dimensional feature vector can be a fine texture, shape, motion pattern change, etc. in the image.

[0090] Please refer to Figure 3 , Figure 3 as Figure 1 the detailed step flowchart of step S103 in

[0091] As​​​​Figure 3 In some embodiments, the step S103 of performing hierarchical clustering on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data can include the following steps S301-S303.

[0092] The step S301 of performing frame-level clustering on the visual data based on the multi-dimensional feature vector to obtain a static visual clustering cluster of the visual data; the plurality of clustering clusters of the visual data include the static visual clustering cluster.

[0093] When the terminal device performs hierarchical clustering on the visual data based on the target feature vector of the visual data, the terminal device can first perform frame-level clustering on the visual data based on the multi-dimensional feature vector of each frame of the visual data, thereby obtaining a static visual clustering cluster of the visual data. In this way, the terminal device can take the static visual clustering cluster as part of the plurality of clustering clusters of the visual data.

[0094] In some embodiments, when the terminal device performs frame-level clustering on the visual data based on the multi-dimensional feature vector of each frame of the visual data, the terminal device can group frames with similar features in the visual data into the same cluster by similarity measurement, i.e., each frame in the visual data is represented by its multi-dimensional feature vector, and unsupervised clustering is performed on these multi-dimensional feature vectors by an unsupervised clustering algorithm, thereby obtaining cluster centers and cluster division results, and finally according to the cluster results, frames belonging to the same cluster are grouped into a group, thereby forming a static visual clustering cluster of the visual data.

[0095] In some embodiments, the step of performing frame-level clustering on the visual data based on the multi-dimensional feature vector in the step S301 can include the following steps: performing similarity measurement on the multi-dimensional feature vector of each frame of the visual data to obtain a similarity measurement result; performing unsupervised clustering on the multi-dimensional feature vector based on the similarity measurement result to obtain a cluster division result of the multi-dimensional feature vector; the cluster division result includes at least one feature clustering cluster; performing cluster matching on each frame of the visual data based on the at least one feature clustering cluster to obtain at least one static visual clustering cluster of the visual data.

[0096] The terminal device first performs similarity measurement processing on the multi-dimensional feature vector of each frame of the visual data to obtain a similarity measurement result. The similarity measurement result can be a similarity measurement result between each frame of image in the visual data relative to an initial cluster center, which can be a multi-dimensional feature vector of an image frame randomly specified by the terminal device in advance in the multi-frame image of the visual data.

[0097] Then, the terminal device performs unsupervised clustering processing on the multi-dimensional feature vector of each frame of image based on the similarity measurement result between each frame of image relative to the initial cluster center as a standard, and updates the cluster center in the process, thereby obtaining the cluster center of the multi-dimensional feature vector and the cluster division result. The cluster division result can include one or more feature clustering clusters.

[0098] Finally, the terminal device matches the multi-dimensional feature vector of each frame of the visual data with the feature clustering cluster in the cluster division result, thereby dividing all frames belonging to the same feature clustering cluster in the visual data into a group, and obtaining one or more static visual clustering clusters of the video data.

[0099] In some embodiments, when the terminal device performs unsupervised clustering processing on the multi-dimensional feature vector of each frame of image to group frames with similar features into the same feature clustering cluster through similarity measurement, the terminal device can first perform initialization processing on the multi-dimensional feature vector to determine the initial cluster center (such as randomly selecting part of the feature vectors as the initial center, or generating the initial center based on the preset cluster number), then calculate the similarity value of the multi-dimensional feature vector of each frame with each initial cluster center based on the preset similarity measurement method (such as Euclidean distance, cosine similarity, etc.), and assign the multi-dimensional feature vector of each frame to the cluster corresponding to the initial cluster center most similar to it according to the similarity value to form a preliminary cluster division, and then, for each cluster after the preliminary cluster division, re-calculate the cluster center based on all multi-dimensional feature vectors in the cluster (such as updating by mean, median, etc. aggregation method) to obtain the updated cluster center, and finally, determine whether the difference between the current updated cluster center and the last round cluster center meets the preset stopping condition (such as the difference being less than a threshold or the number of iterations reaching an upper limit), if not, re-calculate the similarity based on the updated cluster center and repeat the division and update process until the preset condition is met, if yes, output the final cluster center and the corresponding cluster division result.

[0100] In some embodiments, when the terminal device performs cluster matching processing on each frame of the video data based on the feature clusters, the terminal device can parse the cluster division result of the multi-dimensional feature vector, extract the frame unique identifier (such as frame ID, index) corresponding to each feature cluster, form a "cluster ID-frame identifier" mapping relationship table, and then filter all frames corresponding to each cluster from all frames of the visual data based on the mapping relationship table, aggregate each cluster into a cluster-specific frame set, and associate each frame set with the cluster center of the multi-dimensional feature vector corresponding to the frame set, to form structured data including "cluster ID, frame set, and cluster center", that is, a static visual cluster.

[0101] In some embodiments, the terminal device can use the Euclidean distance as the similarity measurement method: .

[0102] wherein, is the multi-dimensional feature vector of the i-th frame; is the center of the j-th cluster; and n is the dimension of the feature vector.

[0103] In addition, the formula for updating the cluster center by the terminal device can be as follows: .

[0104] wherein, is the sample set of the j-th cluster in the t-th iteration; is the number of samples in the cluster .

[0105] Further, the terminal device can calculate the sum of squared errors as the difference between the updated cluster center and the last round cluster center: .

[0106] wherein, k is the number of clusters; is the sample set of the j-th cluster.

[0107] Step S302: performing segmented fusion processing on the multi-dimensional feature vector based on at least one time segment of the visual data, to obtain a frame feature vector fusion result of the at least one time segment.

[0108] When the terminal device performs hierarchical clustering processing on the visual data based on the target feature vector of the visual data, the terminal device can first perform segmented fusion processing on the multi-dimensional feature vector of each frame of the visual data based on at least one time segment of the visual data, that is, aggregate the multi-dimensional feature vectors of all frames in a time segment of the visual data to form a time sequence feature representation at the segment level, to obtain a frame feature vector fusion result of the at least one time segment.

[0109] Step S303: performing a segment-level clustering processing on the visual data based on the frame feature vector fusion result, to obtain a dynamic event clustering cluster of the visual data; the multiple clustering clusters of the visual data include the dynamic event clustering cluster.

[0110] After obtaining the frame feature vector fusion result of each time segment of the visual data, the terminal device performs a clustering analysis on the frame feature vector fusion result, and aggregates the time sequence information between each time segment, so as to realize the segment-level clustering processing on the visual data, and obtain a dynamic event clustering cluster of the visual data. In this way, the terminal device can also regard the dynamic event clustering cluster as a part of the multiple clustering clusters of the visual data.

[0111] In some embodiments, the terminal device can divide the visual data (video / sequence) into multiple time segments through time window division, then perform a clustering analysis on the frame-level features (multi-dimensional feature vectors of each frame of the visual data) within the segments, and aggregate the time sequence information between each time segment to form a dynamic event clustering cluster. In this way, the evolution rule of the event in the time dimension can be captured, which is suitable for video anomaly detection, behavior recognition and the like.

[0112] In some embodiments, the step S303 of "performing a segment-level clustering processing on the visual data based on the frame feature vector fusion result" can include the following steps: performing a clustering analysis on the frame feature vector fusion result, to obtain cluster information of each frame within each time segment of the visual data; aggregating the cluster information based on the time sequence relationship of each time segment of the visual data, to obtain a dynamic event clustering cluster of the visual data.

[0113] When the terminal device performs a segment-level clustering processing on the visual data based on the frame feature vector fusion result of each time segment of the visual data, the terminal device can first perform a clustering analysis on the frame feature vector fusion result, to obtain cluster information of each frame within each time segment of the visual data, and then aggregate the cluster information based on the time sequence relationship of each time segment of the visual data, to obtain a dynamic event clustering cluster of the visual data.

[0114] For example, after the terminal device divides the visual data into multiple time segments, the terminal device performs a clustering analysis on each segment Using a clustering algorithm to obtain a cluster center: }, and a frame cluster label: . Then, the terminal device aggregates the cluster information using a time decay factor: .

[0115] wherein, identifying the number of frames of the cluster k in the segment n.

[0116] Finally, the terminal device performs final clustering on the features within each segment according to the aggregated cluster information ClusterScore:

[0117] wherein, is a time position weight.

[0118] In this embodiment, the terminal device performs frame-level clustering processing on the visual data based on the multi-dimensional feature vector of each frame of the visual data, thereby obtaining a static visual clustering cluster of the visual data. The multi-dimensional feature vector is segmented and fused based on at least one time segment of the visual data, thereby obtaining a frame feature vector fusion result of the at least one time segment. The visual data is subjected to segment-level clustering processing based on the frame feature vector fusion result, thereby obtaining a dynamic event clustering cluster of the visual data. In this way, by adopting a hierarchical clustering strategy combined with general features, dynamic event sequences and subtle static features can be recognized, and deep semantic understanding of the visual data by the system can be achieved. In addition, the hierarchical clustering strategy can also understand the visual data from different perspectives, thereby achieving fine-grained recognition and unknown event discovery of the visual data. The frame-level clustering of the visual data can discover similar static elements or key states (such as “the moment when the car door handle is pulled”, “tires at different angles”), and the segment-level clustering of the visual data can discover similar dynamic event patterns (such as “one person walking around the car”, “two cars meeting”, “an electric car passing by”). In this way, the frame-level clustering and the segment-level clustering of the visual data can automatically induce a large amount of visual data into various fine-grained event categories, and complex scenes can be accurately recognized and unknown new events can be automatically discovered without manual pre-definition.

[0119] In some embodiments, the implicit identification of the clustering cluster includes a machine semantic label.

[0120] It should be noted that the machine semantic label can be obtained by the terminal device by querying an external multi-modal model (or a keyword library) based on the cluster center feature, and the semantic label of the clustering cluster is assigned.

[0121] For reference Figure 4 , Figure 4 The step flowchart of the visual data management method of the vehicle sentry mode provided in the embodiments of the present application is shown in the following figure.

[0122] As Figure 4As shown, in some embodiments, the vehicle sentry mode visual data management method provided by the embodiments of the present application can further include steps S401 and S402 as shown below.

[0123] Step S401: performing aggregation calculation on the target feature vectors in the target cluster based on an aggregation function to obtain a core feature vector of the target cluster; the target cluster is any one of the plurality of clusters.

[0124] It should be noted that the aggregation function can be mean, weighted average, median and / or other forms.

[0125] After the terminal device performs hierarchical clustering processing on the visual data to obtain a plurality of clusters, it respectively takes the plurality of clusters as target clusters, and then for any target cluster, it performs aggregation calculation on the target feature vectors in the target cluster using an aggregation function, thereby obtaining a core feature vector of the target cluster.

[0126] In some embodiments, the terminal device can extract the commonality of the target feature vectors in the target cluster through the aggregation function, thereby obtaining the "representative feature" of the target cluster as the core feature vector.

[0127] Step S402: performing semantic translation on the core feature vector based on a preset visual language large model to obtain a machine semantic label of the target cluster.

[0128] After the terminal device obtains the core feature vector of the target cluster, it can map the core feature vector to an implicit label of the target cluster, that is, perform semantic translation on the core feature vector based on a visual language model (Vision-Language Model, VLM) to generate a machine semantic label of the target cluster.

[0129] In some embodiments, the terminal device can also map the visual features (core feature vectors of the target clusters) to the language feature space and perform semantic alignment through the semantic alignment of the multi-modal large model (such as the visual language model VLM) to generate an implicit label (machine semantic label) for each cluster class. For example, the terminal device maps the visual features to the language feature space through training a multi-modal large model, and ensures that both express similar semantic content in this common space. For a given data set, it is first divided into multiple cluster classes (clusters) through a clustering algorithm. Then, an implicit label is generated for each cluster class, and this process usually relies on the mapping relationship between the features extracted from the visual data and the language description.

[0130] Exemplarily, assume that denotes a set of visual feature vectors (i.e. target feature vectors within a target cluster), representing a corresponding set of textual descriptions (language features). A multi-modal model is used to map visual features to language feature space: .

[0131] wherein, is the mapped visual feature representation, the position in the language feature space.

[0132] Then, these mapped visual features are mapped to k cluster classes . That is, the implicit label of each cluster class can be computed by an aggregation function (mean, weighted mean or other forms of summary): .

[0133] wherein, denotes a certain data point within a cluster , and denotes the number of data points in a cluster .

[0134] Please refer to Figure 5 , Figure 5 for the detailed step flowchart of step S104. Figure 1 As shown in

[0135] , in some embodiments, the above-mentioned step S104: based on the implicit identification of each of the plurality of cluster classes, establishing a semantic index database to perform life cycle management on the visual data, can include the following steps S501 and S502. Figure 5 Step S501: establish an association between the implicit identification of each of the plurality of cluster classes and the metadata of the visual data, to obtain a semantic index database; the metadata includes segment identification data of the visual data, mapping relationship between the visual data and the plurality of cluster classes, and original storage location data of the visual data.

[0136]

[0137] ​​It should be noted that the metadata of the visual data can be metadata of each segment of the visual data. That is, the metadata can include segment identification data (segment unique ID) of each segment of the visual data, mapping relationships between each segment of the visual data and a plurality of clustering clusters (cluster identification ID to which each segment belongs), and original storage location data (original data storage location pointer - storage path, uniform resource locator URL, etc.) of each segment of the visual data.

[0138] When the terminal device establishes the semantic index database of the visual data based on the implicit identifiers of the plurality of clustering clusters, the terminal device can establish an association relationship between the implicit identifiers of the plurality of clustering clusters and the metadata of each segment of the visual data, thereby obtaining the semantic index database.

[0139] In some embodiments, when the terminal device establishes the semantic index database of the visual data, the terminal device can first define a creation schema of the semantic index database based on the clustering cluster, the implicit identifier of the clustering cluster, and attribute information of the metadata of each segment of the visual data, that is, based on the data type of the clustering cluster ID, the format of the machine semantic label, and the specification of the metadata of the video segment (such as the naming rule of the video segment unique ID, the format of the original data storage pointer - path / URL / hash value), the field format, data type, and association rule of the semantic index database are determined, and a schema definition document of the semantic index database is obtained. Then, the terminal device takes the schema definition document as a storage template, and persistently stores the implicit identifier of the clustering cluster and the metadata of each segment of the visual data according to the schema structure, thereby realizing the chain association of "video segment unique ID→cluster ID→machine semantic label→original data pointer" (such as the table structure in the relational database, or the document collection in the document type database), and thereby establishing the semantic index database of the visual data.

[0140] Step S502: storing the visual data in a warehouse based on the semantic index database, and performing semantic retrieval on the visual data stored in the warehouse based on the semantic index database.

[0141] After the terminal device establishes the semantic index database of the visual data, the terminal device can intelligently store each segment of the visual data in a warehouse based on the semantic index database, and in the case that a query request initiated by a user for the visual data is received, the terminal device responds to the query request to perform semantic retrieval on the visual data stored in the warehouse based on the semantic index database.

[0142] In some embodiments, the terminal device can store the video segments with high relevance (e.g., same cluster data) in the same physical or logical "data bucket" based on the semantic index database and the relevance between each segment of visual data and the cluster (i.e., high relevance), thereby achieving intelligent storage of visual data in buckets. In this case, each physical or logical "data bucket" corresponds to one or a group of relevant clusters (usually the same cluster), and the bucket stores the original data of all video segments of the cluster. In addition, the terminal device can also form a mapping relationship between the "data bucket" and the cluster ID at the same time, which can be stored in the semantic index database or stored independently.

[0143] In this embodiment, the terminal device assigns machine semantic labels as implicit identifiers to multiple cluster groups of visual data, and then establishes a semantic index database based on the implicit identifiers of the multiple cluster groups to intelligently store the visual data in buckets, so that the visual data of similar events can be stored in a centralized manner. Not only is it convenient to manage, but it can also greatly optimize the application efficiency of the visual data. For example, when an algorithm team needs a large amount of "vehicle collision" data to optimize the vehicle automatic emergency braking (AEB) algorithm, high-quality positive samples can be directly extracted from the "collision event bucket" in the semantic index database, without the need for manual data collection, which greatly improves the data utilization efficiency and the speed of value mining.

[0144] Please refer to Figure 6 , Figure 6 for Figure 5 the detailed step flowchart of step S502.

[0145] As Figure 6 shown, in some embodiments, the step of "performing semantic retrieval on the visual data stored in buckets based on the semantic index database" in step S502 can include steps S601 and S602 as shown below.

[0146] Step S601: Perform semantic similarity calculation on the query intent information corresponding to the visual data query statement and the visual data stored in the buckets in the semantic index database to obtain a semantic similarity calculation result.

[0147] It should be noted that the visual data query statement can be a query request initiated by the user for the visual data. For example, the user can input a query request (such as "all events involving electric car scratching the left door on x year x month x day") to the terminal device through a human-computer interaction interface (such as a user graphical interface) based on the actual use demand for the visual data. In this way, the terminal device can receive the query request and take the text content therein as a visual data query statement for querying the visual data stored in the semantic index database.

[0148] In addition, the semantic similarity between the query intent information calculated by the terminal device and the visual data can be a cosine similarity.

[0149] After the terminal device stores the visual data in the semantic index database, the terminal device can accurately match and search the user query through semantic retrieval, so as to return the index video and associated information highly related to the query intent to the user in response to the user query. Based on this, after receiving the visual data query statement, the terminal device can first analyze the visual data query statement in depth to obtain the query intent information corresponding to the visual data query statement, and then perform semantic similarity calculation on the query intent information and the visual data stored in the semantic index database to obtain a semantic similarity calculation result. Among them, the semantic similarity calculation result can include the semantic similarity between the query intent information and each segment (or all segments in each clustering cluster) of the visual data.

[0150] Step S602: In a case where the semantic similarity calculation result indicates that the semantic similarity between the query intent information and the target visual data is greater than or equal to a preset threshold, output the target visual data as a query result of the visual data query statement; the target visual data is at least one segment of the visual data stored in the semantic index database.

[0151] It should be noted that the target visual data is a segment of the visual data stored in the semantic index database, or all segments in a clustering cluster of the visual data stored in the semantic index database.

[0152] After obtaining the semantic similarity calculation result between the query intent information and the visual data, if the result indicates that the semantic similarity between the query intent information and the target visual data in the semantic index database is greater than or equal to a preset threshold, the terminal device determines the target video data as visual data (which can include video segments, timestamps, labels, metadata, etc.) highly related to the query intent, and outputs the target visual data as a query result of the visual data query statement.

[0153] In some embodiments, the terminal device, when outputting the target visual data as a query result, can also output relevant recommended content (such as video clips in similar scenarios, different perspective data of the same event, etc.) as part of the query result.

[0154] In this embodiment, after the terminal device stores the visual data in the semantic index database through intelligent warehouse storage, the user query can be accurately matched and searched through semantic retrieval, so as to return the index video and associated information highly related to the query intent to the user in response to the user query. In this way, by using semantic indexing and intelligent warehouse, unstructured visual data is converted into a structured information library, so as to support natural language-level second-speed retrieval of "find all right side scraping events", access and extract the required high-value scene data. Moreover, by understanding the content of each video clip of the visual data through the semantic index database, the user can directly use "find all 'try to pull the door' events" or "find all videos related to 'pet close'" and other high-level semantics to query, and the system quickly locates the relevant "data warehouse" to realize millisecond-level retrieval, thereby effectively solving the problem that the traditional solution cannot efficiently manage and retrieve visual data.

[0155] Next, a complete embodiment of the visual data management method of the vehicle sentry mode provided by the embodiment of the present application is proposed.

[0156] Please refer to Figure 7 , Figure 7 The visual data management method of the vehicle sentry mode provided by the embodiment of the present application is shown in the system overall flowchart of a complete embodiment.

[0157] As shown in Figure 7 , the visual data management method of the vehicle sentry mode provided by the embodiment of the present application can include the steps S701 to S705 as shown below.

[0158] Step S701: data preprocessing for original video: including video segmentation, key frame sampling; feature extraction through self-supervised pre-training model, output high-density feature vector. This process aims to convert the original, unstructured video data into structured, machine-readable high-density feature vector sequence. These feature vectors can efficiently represent the visual and temporal information of the video, providing a strong foundation for downstream tasks such as behavior recognition, video retrieval, content analysis, etc. The whole process mainly includes two core stages: data preprocessing and feature extraction.

[0159] It should be noted that the original video can be the above-mentioned visual data, the self-supervised pre-training model can be the above-mentioned self-supervised feature extraction model, and the high-density feature vector can be the target feature vector of the above-mentioned visual data.

[0160] In addition, the specific operations of data preprocessing and feature extraction are the same as the operations of the terminal device in the above embodiment for performing preprocessing on visual data and feature extraction based on a self-supervised feature extraction model. Here, the same content will not be described again.

[0161] Step S702: Frame-level clustering calculation is performed, the feature vectors of all frames are clustered, and a static visual cluster clustering result is output. That is, frames with similar features are classified into the same cluster through similarity measurement, the goal is to represent each frame in the video as a feature vector, and the cluster center and cluster division result are obtained through an unsupervised feature vector clustering algorithm; according to the cluster result, frames belonging to the same cluster are classified into a group to form a static visual cluster.

[0162] It should be noted that the static visual cluster is the above-mentioned static visual clustering cluster. Here, the specific operation of frame-level clustering is the same as the operation of the terminal device in the above embodiment for performing frame-level clustering processing on visual data. Here, the same content will not be described again.

[0163] Step S703: Segment-level temporal feature fusion is performed, cluster information of all frames in the segment time period is aggregated, and a dynamic event cluster clustering result is output. That is, the video / sequence is divided into multiple time segments through time window division, frame-level features are clustered and analyzed in the segment, and dynamic event clusters are aggregated using temporal information.

[0164] It should be noted that the dynamic event cluster is the above-mentioned dynamic event clustering cluster. Here, the specific operation of segment-level clustering is the same as the operation of the terminal device in the above embodiment for performing segment-level clustering processing on visual data. Here, the same content will not be described again.

[0165] Step S704: Through semantic alignment of the multi-modal large model, the visual features are mapped to the language feature space and semantically aligned, and an implicit label is generated for each cluster; store data, store the mapping relationship of cluster labels and data pointers. That is, each cluster (Cluster) generated after clustering is assigned a machine-generated semantic label. Subsequently, instead of simply storing the original video according to time or vehicle ID, a semantic index database is established, through which the unique ID of each video segment, the cluster ID to which it belongs, the machine semantic label, and the pointer to the original data storage location are recorded. Thus, physically, video segments with high correlation (cluster data) are stored in the same physical or logical "data warehouse" (DataBucket), realizing "intelligent warehousing".

[0166] It should be noted that the implicit label can be the implicit identification described above. Here, generating the implicit label and establishing the semantic index database to intelligently store the video data are the same as the operations of generating the implicit identification for the clustering cluster based on the visual language large model and establishing the semantic index database based on the implicit identification to store the visual data in the warehouse, and details are not repeated for the same content.

[0167] Step S705: Through semantic retrieval technology, accurate matching and search of the user query are realized. If a result highly related to the query intention is retrieved, the corresponding index video and associated information are returned. That is, by deeply analyzing the query statement, the real intention and context of the user are understood, so that the content with similar semantics in the vector space is found, and the semantic similarity (such as cosine similarity) between the query and the candidate content is calculated and a threshold is set to realize it. If the similarity exceeds the preset threshold, the system returns the most matching index video (which may include video clips, time stamps, labels, metadata, etc.), and the returned result may be accompanied by the recommendation of related content (such as similar scenes, different perspectives of the same event, etc.).

[0168] In this embodiment, through self-supervised learning feature extraction and unsupervised hierarchical clustering, the self-supervised pre-training model learns from data autonomously without any manual annotation, realizing an end-to-end automatic pipeline from feature extraction to event classification. Through unsupervised clustering, the internal mode of data is automatically discovered, unknown new event types can be autonomously induced and discovered, and the system has the ability of self-evolution and continuous learning. In addition, the hierarchical clustering strategy combined with general features can identify dynamic event sequences and subtle static features, realizing deep semantic understanding of “who, what, what, and why”. Furthermore, the unstructured data is converted into a structured information library through semantic indexing and intelligent storage, so as to support natural language level second-speed retrieval, and enable users to efficiently access and extract the required high-value scene data. Therefore, this embodiment constructs an automatic, intelligent, fine-grained and scalable mass vehicle sentinel visual data understanding and management system, which overcomes the inherent defects of traditional solutions based on manual or rules in efficiency, objectivity, granularity and flexibility, so as to convert the original video data into a truly structured information asset that can be efficiently utilized.

[0169] Please refer to Figure 8 The embodiment of the application also provides a vehicle sentinel mode visual data management device.

[0170] The vehicle sentinel mode visual data management device provided by the embodiment of the application can realize the vehicle sentinel mode visual data management method described above.

[0171] As Figure 8As shown, the vehicle sentry mode visual data management device provided by the embodiments of the present application can include: an acquisition module configured to acquire visual data in a vehicle sentry mode; a self-supervised feature extraction module configured to perform feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data, wherein the self-supervised feature extraction model is obtained based on self-supervised learning training of a general visual feature extraction model on unannotated sentry visual sample data; a hierarchical clustering module configured to perform hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data; a semantic indexing module configured to establish a semantic indexing database based on implicit identifiers of the plurality of clustering clusters to perform life cycle management on the visual data.

[0172] In some embodiments, the target feature vector includes a multi-dimensional feature vector of each frame of the visual data. The hierarchical clustering module is further configured to perform frame-level clustering processing on the visual data based on the multi-dimensional feature vector to obtain a static visual clustering cluster of the visual data; the plurality of clustering clusters of the visual data includes the static visual clustering cluster; perform segment fusion processing on the multi-dimensional feature vector based on at least one time segment of the visual data to obtain a frame feature vector fusion result of the at least one time segment; perform segment-level clustering processing on the visual data based on the frame feature vector fusion result to obtain a dynamic event clustering cluster of the visual data; and the plurality of clustering clusters of the visual data includes the dynamic event clustering cluster.

[0173] In some embodiments, the hierarchical clustering module is further configured to perform similarity measurement processing on the multi-dimensional feature vector of each frame of the visual data to obtain a similarity measurement result; perform unsupervised clustering processing on the multi-dimensional feature vector based on the similarity measurement result to obtain a cluster division result of the multi-dimensional feature vector; the cluster division result includes at least one feature clustering cluster; and perform cluster matching processing on each frame of the visual data based on the at least one feature clustering cluster to obtain at least one static visual clustering cluster of the visual data.

[0174] In some embodiments, the hierarchical clustering module is further configured to perform clustering analysis on the frame feature vector fusion result to obtain cluster information of each frame within each time segment of the visual data; and aggregate the cluster information based on a time sequence relationship of each time segment of the visual data to obtain a dynamic event clustering cluster of the visual data.

[0175] In some embodiments, the implicit identifier comprises a machine semantic label, and the vehicle sentry mode visual data management apparatus provided by the embodiments of the present application further comprises: an implicit label generation module configured to: perform aggregation calculation on a target feature vector in a target cluster based on an aggregation function to obtain a core feature vector of the target cluster; the target cluster is any one of the plurality of clusters; and perform semantic conversion on the core feature vector based on a preset visual language large model to obtain a machine semantic label of the target cluster.

[0176] In some embodiments, the semantic index module is further configured to: establish an association between the implicit identifier of each of the plurality of clusters and metadata of the visual data to obtain a semantic index database; the metadata comprises segment identifier data of the visual data, a mapping relationship between the visual data and the plurality of clusters, and original storage location data of the visual data; store the visual data in a warehouse based on the semantic index database, and perform semantic retrieval on the visual data stored in the warehouse based on the semantic index database.

[0177] In some embodiments, the semantic index module is further configured to: perform semantic similarity calculation on query intent information corresponding to a visual data query statement and the visual data stored in the warehouse based on the semantic index database to obtain a semantic similarity calculation result; and in a case where the semantic similarity calculation result indicates that the semantic similarity between the query intent information and target visual data is greater than or equal to a preset threshold, output the target visual data as a query result of the visual data query statement; the target visual data is at least one segment of the visual data stored in the warehouse based on the semantic index database.

[0178] In some embodiments, the acquisition module is further configured to: perform key frame sampling on a target segment of the visual data to obtain a key frame sequence; the target segment is at least one of the plurality of segments of the visual data; and perform feature extraction processing on at least one key frame in the key frame sequence based on a self-supervised feature extraction model, and fuse the extracted key frame multi-dimensional feature vectors into a multi-dimensional feature vector of the target segment based on the self-supervised feature extraction model to output; the target feature vector of the visual data comprises the multi-dimensional feature vector of the target segment.

[0179] It should be noted that the specific implementation of the vehicle sentry mode visual data management apparatus provided by the embodiments of the present application is basically the same as the specific embodiments of the vehicle sentry mode visual data management method described above, and will not be repeated here.

[0180] Please refer to Figure 9The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the vehicle sentry mode visual data management method when executing the computer program.

[0181] In some embodiments, the electronic device can be a tablet computer, a smart phone, a vehicle-mounted hardware platform (for example, a vehicle-mounted computer, etc.), a wearable device, or any smart terminal.

[0182] As shown in Figure 9 The electronic device provided by the embodiment of the present application can comprise: The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiment of the present application. The memory 902 can be implemented in the form of a read-only memory (ReadOnly Memory, ROM), a static storage device, a dynamic storage device, or a random access memory (Random Access Memory, RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the vehicle sentry mode visual data management method of the embodiment of the present application. The input / output interface 903 is used to realize information input and output. The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 905 is used to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device. The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize communication connection between them in the device.

[0183] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the vehicle sentry mode visual data management method.

[0184] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0185] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements steps substantially the same as the specific embodiments of the vehicle sentry mode visual data management method described above, and thus will not be described herein again.

[0186] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0187] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0188] The system embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments.

[0189] Those skilled in the art can understand that all or some steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0190] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "first" can be termed as "second", "second" can be termed as "first". It is also to be understood that the use of the terms "and", "or", and "both" and "and / or" as used herein, can include a variety of meanings that also is expected to enable the proper understanding of the embodiment of the application as claimed. It is also to be understood that the use of "including", "comprising", or "having" of any described or illustrated elements is not

[0191] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0192] In several embodiments provided by the application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between systems or units, which can be electrical, mechanical or other forms.

[0193] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0194] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0195] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0196] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for visual data management of a vehicle sentry mode, characterized by, The method comprises: acquiring visual data in a vehicle sentry mode; performing feature extraction processing on the visual data based on a self-supervised feature extraction model to obtain a target feature vector of the visual data; the self-supervised feature extraction model is obtained based on a general visual feature extraction model through self-supervised learning training on unlabelled sentry visual sample data; performing hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data; based on the implicit identification of each of the plurality of clustering clusters, establishing a semantic index database to manage the life cycle of the visual data.

2. The method of claim 1, wherein, The target feature vector comprises a multi-dimensional feature vector of each frame of the visual data; The hierarchical clustering processing on the visual data based on the target feature vector to obtain a plurality of clustering clusters of the visual data comprises: performing frame-level clustering processing on the visual data based on the multi-dimensional feature vector to obtain a static visual clustering cluster of the visual data; the plurality of clustering clusters of the visual data comprises the static visual clustering cluster; performing segmentation fusion processing on the multi-dimensional feature vector based on at least one time segment of the visual data to obtain a frame feature vector fusion result of the at least one time segment; performing segment-level clustering processing on the visual data based on the frame feature vector fusion result to obtain a dynamic event clustering cluster of the visual data; the plurality of clustering clusters of the visual data comprises the dynamic event clustering cluster.

3. The method of claim 2, wherein, The frame-level clustering processing on the visual data based on the multi-dimensional feature vector comprises: performing similarity measurement processing on the multi-dimensional feature vector of each frame of the visual data to obtain a similarity measurement result; performing unsupervised clustering processing on the multi-dimensional feature vector based on the similarity measurement result to obtain a cluster division result of the multi-dimensional feature vector; the cluster division result comprises at least one feature clustering cluster; performing cluster matching processing on each frame of the visual data based on the at least one feature clustering cluster to obtain at least one static visual clustering cluster of the visual data.

4. The method of claim 2, wherein, The segment-level clustering processing on the visual data based on the frame feature vector fusion result comprises: performing clustering analysis on the frame feature vector fusion result to obtain cluster information of each frame within each time segment of the visual data; performing aggregation on the cluster information based on the time sequence relationship of each time segment of the visual data to obtain a dynamic event clustering cluster of the visual data.

5. The method of claim 1, wherein, The implicit identification comprises a machine semantic label, and the method further comprises: performing aggregation calculation on a target feature vector in a target clustering cluster based on an aggregation function to obtain a core feature vector of the target clustering cluster; the target clustering cluster is any one of the plurality of clustering clusters; performing semantic conversion on the core feature vector based on a preset visual language large model to obtain a machine semantic label of the target clustering cluster.

6. The method of claim 1, wherein, The establishment of a semantic index database based on the implicit identification of each of the plurality of clustering clusters to manage the life cycle of the visual data comprises: establishing an association between the implicit identifier of each of the plurality of clustering clusters and metadata of the visual data, to obtain a semantic index database; the metadata includes segment identifier data of the visual data, a mapping relationship between the visual data and the plurality of clustering clusters, and original storage location data of the visual data; based on the semantic index database, the visual data is stored in a warehouse, and based on the semantic index database, the visual data stored in the warehouse is semantically searched.

7. The method of claim 6, wherein, The semantic search of the visual data stored in the warehouse based on the semantic index database comprises: the query intent information corresponding to the visual data query statement is semantically similar to the visual data stored in the warehouse in the semantic index database, and a semantic similarity calculation result is obtained; in the case where the semantic similarity calculation result indicates that the semantic similarity between the query intent information and the target visual data is greater than or equal to a preset threshold, the target visual data is output as the query result of the visual data query statement; the target visual data is at least one segment of the visual data stored in the warehouse in the semantic index database.

8. The method according to any one of claims 1 to 7, characterized in that, The feature extraction processing of the visual data based on the self-supervised feature extraction model comprises: key frame sampling is performed on a target segment of the visual data to obtain a key frame sequence; the target segment is at least one of the plurality of segments of the visual data; based on the self-supervised feature extraction model, at least one key frame in the key frame sequence is subjected to feature extraction processing, and the extracted multi-dimensional feature vector of the key frame is fused into the multi-dimensional feature vector of the target segment and output; the target feature vector of the visual data includes the multi-dimensional feature vector of the target segment.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the vehicle sentinel mode visual data management method of any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the vehicle sentinel mode visual data management method of any one of claims 1-8.

11. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the vehicle sentinel mode visual data management method of any one of claims 1-8.

Images