View data processing method, device, system, equipment, medium and program product

Abstract

The invention discloses a view data processing method, device, system and equipment, a medium and a program product, and the method comprises the steps: pulling a video stream from a data source, and extracting the view data of the video stream; performing structured enhancement processing on the view data by adopting an improved cavity space convolution pyramid pooling model fused with an attention mechanism to obtain view structured data of the corresponding view data; uploading the view structured data to a cloud end through the edge processing node, and performing multivariate heterogeneous data alignment on each view structured data by the cloud end through a multi-modal pre-training neural network model to obtain corresponding view fusion data; according to the method, the structured processing and multivariate heterogeneous data alignment are performed on the view data of the articulated naturality web, so that a high-quality data basis can be provided for view data development and application, and the application development and business operation requirements of the articulated naturality web can be met.

Description

Technical Field

[0001] This invention relates to the field of visual network technology, and in particular to a view data processing method, apparatus, system, device, medium, and program product. Background Technology

[0002] The Video Internet of Things (VIoT) is a new type of video information service based on video communication technology, using IoT (Internet of Things) connectivity as a carrier, and integrating technologies such as cloud computing, communication, AI (Artificial Intelligence), and big data. It provides end-to-end processing capabilities for video acquisition, transmission, storage, analysis, and presentation for various video-related business scenarios such as video surveillance, video conferencing, and video intercom. With the continuous development of multimedia and communication technologies in recent years, hundreds of millions of networked surveillance cameras collect massive amounts of video data daily for different monitoring purposes across various video business scenarios. Faced with this massive amount of video data, how to collect, query, analyze, and understand it has become a crucial area of ​​exploration for the industry's development.

[0003] The video data in the visual network is primarily encoded video data. Part of this video data is stored in the cloud periodically for retrieval and playback; the other part is decoded in real-time at the playback end for real-time monitoring. Currently, the processing and utilization of video data in the visual network mainly fall into two categories: detection and recognition (e.g., detecting and recognizing video data to generate analysis results) and information sharing (e.g., achieving information sharing and interoperability of video data). This approach cannot meet the needs of visual network application development and business operations. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a view data processing method, apparatus, system, device, medium, and program product. By performing structured processing and multi-dimensional heterogeneous data alignment on the view data of the visual network, a high-quality data foundation can be provided for the development and application of view data, thereby meeting the needs of visual network application development and business operation.

[0005] In a first aspect, embodiments of the present invention provide a view data processing method, applied to an edge processing node or a data acquisition terminal, comprising:

[0006] Pull the video stream from the data source and extract the view data of the video stream;

[0007] The view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structure data of the corresponding view data.

[0008] The view structured data is uploaded to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0009] As an improvement to the above scheme, the view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism to obtain the corresponding view structured data, including:

[0010] Save the extracted view data to the data storage module;

[0011] New view data is read from the data storage module in a polling manner according to a preset first time interval, and the new view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data.

[0012] The structured data of the view is saved to the data storage module.

[0013] As an improvement to the above solution, the step of pulling the video stream from the data source includes:

[0014] Poll the data source to detect the video streaming protocols it supports;

[0015] The video stream is retrieved from the data source according to the video streaming protocol supported by the data source.

[0016] As an improvement to the above solution, the extraction of view data from the video stream includes:

[0017] The video stream is subjected to keyframe extraction using an inter-frame difference algorithm based on fusion clustering analysis to obtain the view data of the video stream.

[0018] As an improvement to the above scheme, the step of using an inter-frame difference algorithm based on fusion clustering analysis to extract keyframes from the video stream to obtain view data of the video stream includes:

[0019] Video keyframes are extracted from the video stream according to the frame extraction time interval.

[0020] Cluster analysis is performed on the extracted video keyframes to filter out scene frames in the video keyframes, and a first candidate keyframe set is obtained.

[0021] Inter-frame difference processing is performed on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream.

[0022] As an improvement to the above scheme, the step of extracting video keyframes from the video stream according to the frame extraction time interval includes:

[0023] The video stream is sliced ​​into TS segments according to a preset second time interval to obtain multiple TS segment video files;

[0024] The frame extraction time interval of the corresponding TS segment video file is determined based on the duration of the TS segment video file and the video frame rate of the corresponding TS segment video file.

[0025] Video keyframes are extracted from each of the TS slice video files according to the frame extraction time interval.

[0026] As an improvement to the above scheme, the step of performing inter-frame difference processing on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream includes:

[0027] Convert the video keyframes in the first candidate keyframe set from the RGB color space to the LUV color space;

[0028] In the LUV color space, the average difference intensity between several consecutive video keyframes in the first candidate keyframe set is calculated, and the video keyframe with the largest average difference intensity among several consecutive video keyframes is extracted and saved to the second candidate keyframe set.

[0029] Extract the first video keyframe from the second candidate keyframe set into the keyframe set;

[0030] Determine whether the distance between the next video keyframe in the second candidate keyframe set and any video keyframe in the keyframe set is less than a preset inter-frame distance threshold.

[0031] If so, discard the next video keyframe;

[0032] If not, extract the next video keyframe to the keyframe set;

[0033] The video keyframes in the keyframe set are view data from the corresponding data source.

[0034] As an improvement to the above scheme, the improved dilated spatial convolution pyramid pooling model that integrates the attention mechanism includes a backbone network, an improved dilated spatial convolution pyramid pooling module, an attention mechanism module, a target feature extraction network, and a graph convolutional network.

[0035] The process involves polling and reading newly added view data from the data storage module at preset first time intervals, and then performing structure enhancement processing on the newly added view data using an improved dilated spatial convolution pyramid pooling model with fused attention mechanism to obtain the corresponding view structured data, including:

[0036] The backbone network is used to extract basic features from the newly added view data to obtain a first feature map representing high-level semantics.

[0037] The improved dilated spatial convolution pyramid pooling module is used to extract multi-scale information from the first feature map;

[0038] The multi-scale information is fused, and the fused multi-scale information is input into the attention mechanism module to obtain a scene semantic feature map;

[0039] The target feature map is obtained by extracting target features from the first feature map using the target feature extraction network.

[0040] The scene semantic feature map and the target feature map are input into the graph convolutional network to obtain semantic feature information;

[0041] The semantic feature information is processed in a structured manner to obtain the view structured data.

[0042] Secondly, embodiments of the present invention provide a view data processing method applied in the cloud, comprising:

[0043] The view structured data from multiple data sources uploaded by edge processing nodes is aggregated and stored; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism;

[0044] By using a multimodal pre-trained neural network model to align the structured data of each view with heterogeneous data, corresponding view fusion data is obtained.

[0045] The data from each of the aforementioned view fusions is categorized and stored.

[0046] As an improvement to the above scheme, the multimodal pre-trained neural network model includes a view encoder, a contrastive learning module, and a multimodal fusion module;

[0047] The step of aligning the structured data of each view with heterogeneous data using a multimodal pre-trained neural network model to obtain corresponding view fusion data includes:

[0048] For each of the view structured data, the view encoder extracts multimodal feature vectors from the view structured data to obtain the text features and image features of the corresponding view structured data;

[0049] The contrastive learning module performs semantic alignment between the text features and the image features to generate sequence features.

[0050] The sequence features are fused using the multimodal fusion module to obtain corresponding view fusion data; the view fusion data includes multimodal fusion feature values ​​obtained by deep fusion of the text features and the image features.

[0051] As an improvement to the above solution, the classification and storage of the fused view data includes:

[0052] The view fusion data is categorized and databases are built according to preset classification rules to obtain multiple databases.

[0053] As an improvement to the above solution, the method further includes:

[0054] By performing data standard mapping and alignment on the label values ​​of the same data attribute in the structured data of the views from different data sources, the standard label values ​​of the corresponding view fusion data are obtained.

[0055] Based on the standard label values ​​of the view fusion data, a query statement is generated for querying the view structured data of the same data attribute from multiple databases.

[0056] As an improvement to the above solution, the method further includes:

[0057] Receive structured view data from multiple data sources uploaded by edge processing nodes according to business requirements;

[0058] The structured view data is obtained by the data acquisition terminal after performing structured processing on the view data it collects.

[0059] Alternatively, the structured view data is obtained by edge processing nodes performing structured processing on view data from multiple data sources.

[0060] As an improvement to the above solution, the structured processing of the view data includes:

[0061] Pull the video stream from the data source;

[0062] Extract the view data of the video stream and save the extracted view data to the data storage module;

[0063] New view data is read from the data storage module in a polling manner according to a preset first time interval, and the new view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data.

[0064] Thirdly, embodiments of the present invention provide a view data processing device, applied to an edge processing node or a data acquisition terminal, including: a video access module, an algorithm service module, and a data upload module;

[0065] The video access module is used to pull video streams from the data source;

[0066] The algorithm service module is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism, so as to obtain the view structured data of the corresponding view data.

[0067] The data upload module is used to upload the view structured data to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0068] Fourthly, embodiments of the present invention provide a view data processing device applied in the cloud, comprising: a data integration module, a data calculation module, and a classification storage module;

[0069] The data integration module is used to aggregate and store view structured data from multiple data sources uploaded by edge processing nodes; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism;

[0070] The data calculation module is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0071] The classification and storage module is used to classify and store the fused data of each view.

[0072] Fifthly, embodiments of the present invention provide a view data processing system, including: a cloud, an edge processing node, and a data acquisition terminal; the edge processing node includes: a video access module, an algorithm service module, and a data upload module; the cloud includes: a data integration module, a data calculation module, and a classification storage module;

[0073] The video access module is used to pull video streams from the data sources corresponding to each data acquisition terminal and extract view data of the video streams;

[0074] The algorithm service module is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism, so as to obtain the view structured data of the corresponding view data.

[0075] The data upload module is used to upload the view structured data to the cloud through the edge processing node;

[0076] The data integration module is used to aggregate and store the view structured data from multiple data sources uploaded by edge nodes.

[0077] The data calculation module is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0078] The classification and storage module is used to classify and store the fused data of each view.

[0079] In a sixth aspect, embodiments of the present invention provide a view data processing device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the view data processing method as described in any one of the first aspects or the view data processing method as described in any one of the second aspects.

[0080] In a seventh aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a view data processing method as described in any one of the first aspects or any one of the second aspects.

[0081] Eighthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the view data processing method as described in any one of the first aspects or the view data processing method as described in any one of the second aspects.

[0082] Compared to existing technologies, the present invention provides a view data processing method, apparatus, system, device, medium, and program product. This method involves pulling a video stream from a data source and extracting view data from the video stream; performing structured enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism to obtain structured view data; and uploading the structured view data to the cloud via an edge processing node. The cloud then uses a multimodal pre-trained neural network model to perform heterogeneous data alignment on each of the structured view data sets, resulting in fused view data. This invention, by performing structured processing and heterogeneous data alignment on view data for the visual network, provides a high-quality data foundation for view data development and applications, thereby meeting the needs of visual network application development and business operations. Attached Figure Description

[0083] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of the system architecture of a view data processing system provided in an embodiment of the present invention;

[0085] Figure 2 This is a schematic diagram of the view data structuring process provided in an embodiment of the present invention;

[0086] Figure 3 This is a schematic diagram of the view big data processing and application development process provided in the embodiments of the present invention;

[0087] Figure 4 This is a flowchart of a view data processing method provided in an embodiment of the present invention;

[0088] Figure 5 This is a schematic diagram of the improved dilated spatial convolution pyramid pooling model with fused attention mechanism provided in the embodiments of the present invention;

[0089] Figure 6 This is another flowchart of a view data processing method provided in an embodiment of the present invention;

[0090] Figure 7 This is a structural block diagram of a view data processing device provided in an embodiment of the present invention;

[0091] Figure 8 This is another structural block diagram of a view data processing device provided in an embodiment of the present invention;

[0092] Figure 9 This is a structural block diagram of a view data processing system provided in an embodiment of the present invention;

[0093] Figure 10 This is a structural block diagram of a view data processing device provided in an embodiment of the present invention. Detailed Implementation

[0094] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0095] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0096] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "multiple or several" refer to two or more, and the same applies to "multiple / items or several kinds / items." "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0097] This invention proposes a view data processing system that applies new technologies such as big data and artificial intelligence to the video network. It utilizes video streams retrieved and structured by data acquisition terminals or edge processing nodes (added servers), then uploads and aggregates them to the cloud. The cloud then aggregates, aligns, and stores the structured view data, enabling efficient integration and intelligent processing of the video network's view data. This breaks the limitations of traditional view data transmission and allows for cross-domain integration with other related business data, forming a centralized, aligned, and efficiently searchable system. This provides a high-quality data foundation for view data development and applications, thus meeting the needs of video network application development and business operations.

[0098] It should be noted that the term "view data" can also be described as view big data or video data. These three terms can be used interchangeably and are not specifically limited in this embodiment of the invention.

[0099] The system architecture of the view data processing system is as follows: Figure 1 As shown, the system consists of three parts: cloud, edge processing nodes, and data acquisition terminals.

[0100] Data acquisition end: This includes traditional acquisition devices and new computing devices in the video network. It should be understood that a traditional acquisition device or a new computing device is regarded as a data source for collecting full-sensory data in the video network, including but not limited to video streams, video files, IoT information, etc. Among them, IoT information refers to the real-time acquisition of any object or process that needs to be monitored, connected, or interacted with through various information sensors, radio frequency identification technology, global positioning system, infrared sensors, laser scanners, and other devices and technologies, collecting various required information such as sound, light, heat, electricity, mechanics, chemistry, biology, and location.

[0101] Traditional data acquisition devices include, but are not limited to, traditional cameras, which have image acquisition and encoding capabilities, but are limited by computing power and do not support the deployment of edge-side algorithms for AI analysis and recognition; in this case, the data output by traditional cameras is view data.

[0102] New computing power equipment: AI cameras with certain computing power, and capable of image acquisition, encoding, AI analysis and recognition, can realize structured processing of view data on the edge; at this time, the data output by the AI ​​camera can be view data or view and structured data.

[0103] Edge processing nodes are responsible for aggregating, storing, analyzing, and identifying structured view data / view data from the data acquisition terminals, and uploading and aggregating structured view data from various data sources to the cloud. Edge processing nodes include: a video access module, an algorithm service module, and a data storage module.

[0104] Video access module: pulls video streams from data acquisition devices and extracts key frames of video at certain intervals or according to certain rules, and saves them to the data storage module in image format (i.e., view data);

[0105] Algorithm service module: Detects, analyzes, and structures video keyframes (i.e., view data) using the view data toolchain to obtain structured view data;

[0106] Data storage module: Stores view data (i.e., unstructured view data) and structured view data from various data sources.

[0107] It should be understood that new computing devices can also be equipped with video access modules, algorithm service modules, and data storage modules to perform the formatting processing of the aforementioned view data.

[0108] Cloud-based: Addressing the needs of visual network data applications, it provides end-to-end data development and management, from access, processing, calculation, and analysis to development. The cloud-based system includes: a data integration module, a data calculation module, a categorized storage module, a data analysis module, and a data management module.

[0109] Data integration module: Aggregates and stores all view data or view structured data from the data sources of each edge node to the view big data platform; based on functional and business requirements, it supports aggregating the view structured data uploaded by each edge node from the source type (including but not limited to MySQL, Oracle, Kafka, RestAPI, etc.) into the destination type (including but not limited to Hadoop cluster (Hive), ElasticSearch cluster, Kafka cluster, etc.).

[0110] The categorized storage module primarily uses HDFS distributed files, HBase, data warehouses, and other methods to categorize and store various types of data, such as tag libraries, feature libraries, and view libraries.

[0111] The data computation module preprocesses the structured data of the view, including but not limited to data cleaning, splicing, and metadata mining. Then, it integrates the aggregated view and other structured data through algorithms. The integrated data can be stored in the HDFS file system according to the unified standards and process specifications defined by the data definition, and an access index is built through HBase to support retrieval and querying through query statements.

[0112] Data Analysis Module: Utilizes tools and algorithms such as batch analysis, real-time analysis, interactive analysis, machine learning, and deep learning for data mining and analysis;

[0113] Data Management Module: Support refers to the access and management capabilities provided to external parties for various data resources, including but not limited to data push services, data authentication services, data operation services, comparison and subscription services, etc.

[0114] The Visual Network Big Data Application AS (Application Server) parses task content based on user business settings and system business settings, supports business logic processing for various data applications, and completes the distribution of data application development tasks.

[0115] To meet the needs of video network services, this project implements a complete workflow for accessing, processing, developing, and applying view big data, including two main processes: view data structuring and view big data processing and application development. The view data structuring process is as follows: Figure 2 As shown, the entire process includes a video access module deployed on new computing devices or edge processing nodes pulling video streams from the data acquisition end, processing them into structured view data by the algorithm service module, and then uploading and aggregating them to the cloud. The view big data processing and application development process is divided into two sub-processes, such as... Figure 3 As shown, one is the offline big data aggregation and fusion process, which aggregates the structured data of various edge processing nodes and connects them to the cloud for the entire process of data integration, storage, computation, and management; the other is the big data application development process, which is a process of real-time online data mining and analysis in combination with the big data application needs of the visual network.

[0116] It should be noted that the term "cloud" can also describe a cloud-based big data platform; the two can be used interchangeably and are not specifically limited in this embodiment of the invention. The following provides a detailed description of the view data processing methods involved in the above-described view data structuring process.

[0117] The following section provides a detailed explanation of the view data processing methods involved in the above-mentioned view big data processing and application development process.

[0118] Please see Figure 4 , Figure 4 This is a flowchart of a view data processing method provided in an embodiment of the present invention. The view data processing method, applied to an edge processing node or a data acquisition terminal, includes:

[0119] S11: Pull the video stream from the data source and extract the view data of the video stream;

[0120] S12: The view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structure data of the corresponding view data.

[0121] S13: The view structured data is uploaded to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0122] It should be noted that the view data processing method described in this embodiment of the invention can be executed by a novel computing device in an edge processing node or a data acquisition terminal. When using a novel computing device, the view data structuring process based on an improved dilated spatial convolution pyramid pooling model employing a fusion attention mechanism, as described in steps S11-S12, is first executed by the novel computing device. The resulting structured view data is then sent to the edge processing node and uploaded to the cloud. When using an edge processing node, the edge processing node pulls the currently acquired video stream from the data acquisition terminal and executes the view data structuring process based on an improved dilated spatial convolution pyramid pooling model employing a fusion attention mechanism, as described in steps S11-S12. The resulting structured view data is then directly uploaded to the cloud. Subsequently, the cloud uses a multimodal pre-trained neural network model to perform multi-variable heterogeneous data alignment on each of the structured view data sets to obtain corresponding view fusion data. This invention achieves multi-source heterogeneous data fusion by structuring and aligning the view data of the visual network, providing a high-quality data foundation for view data development and application, thereby meeting the needs of visual network application development and business operation.

[0123] It should be understood that the structured processing of view data performed by the data acquisition terminal is the same as that performed by the edge node. The following will take the structured processing of view data performed by the edge node as an example to explain the structured processing of view data in detail.

[0124] Specifically, the step of pulling the video stream from the data source includes:

[0125] Poll the data source to detect the video streaming protocols it supports;

[0126] The video stream is retrieved from the data source according to the video streaming protocol supported by the data source.

[0127] For example, the video access module deployed on the edge processing node automatically detects the video streaming protocols supported by the data source (which can also be described as the video source) through polling, including but not limited to RTMP, RTSP, RTP, FLV, HLS and other transmission protocols, and pulls the video stream from the data acquisition end based on the supported video streaming protocols.

[0128] Specifically, the extraction of view data from the video stream includes:

[0129] The video stream is subjected to keyframe extraction using an inter-frame difference algorithm based on fusion clustering analysis to obtain the view data of the video stream.

[0130] Specifically, the step of using an inter-frame difference algorithm with fusion clustering analysis to extract keyframes from the video stream to obtain view data of the video stream includes:

[0131] Video keyframes are extracted from the video stream according to the frame extraction time interval.

[0132] Cluster analysis is performed on the extracted video keyframes to filter out scene frames in the video keyframes, and a first candidate keyframe set is obtained.

[0133] Inter-frame difference processing is performed on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream.

[0134] Specifically, the step of extracting keyframes from the video stream according to the frame extraction time interval includes:

[0135] The video stream is sliced ​​into TS (Transport Stream) segments according to a preset second time interval to obtain multiple TS segment video files;

[0136] The frame extraction time interval of the corresponding TS segment video file is determined based on the duration of the TS segment video file and the video frame rate of the corresponding TS segment video file.

[0137] Video keyframes are extracted from each of the TS slice video files according to the frame extraction time interval.

[0138] Specifically, the step of performing inter-frame difference processing on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream includes:

[0139] Convert the video keyframes in the first candidate keyframe set from the RGB color space to the LUV color space;

[0140] In the LUV color space, the average difference intensity between several consecutive video keyframes in the first candidate keyframe set is calculated, and the video keyframe with the largest average difference intensity among several consecutive video keyframes is extracted and saved to the second candidate keyframe set.

[0141] Extract the first video keyframe from the second candidate keyframe set into the keyframe set;

[0142] Determine whether the distance between the next video keyframe in the second candidate keyframe set and any video keyframe in the keyframe set is less than a preset inter-frame distance threshold.

[0143] If so, discard the next video keyframe;

[0144] If not, extract the next video keyframe to the keyframe set;

[0145] The video keyframes in the keyframe set are view data from the corresponding data source.

[0146] Considering the real-time changes in video content, this embodiment of the invention extracts video keyframes by using a video access module to perform inter-frame difference analysis through clustering at certain time intervals on the pulled video stream. This ensures that the extracted video keyframes effectively represent the video content, improves the extraction efficiency, and removes redundant keyframes. The specific video keyframe extraction process is as follows:

[0147] Since video is unstructured data information based on time flow, video content within a short time interval has a high degree of similarity. Therefore, in this embodiment of the invention, the retrieved video is first sliced ​​into TS segments according to a second time interval (e.g., every 5 seconds).

[0148] Based on the duration and frame rate of the TS segment video file, the required frame extraction time interval is determined; for example, the duration of each TS segment video file is divided by its frame rate to obtain the corresponding frame extraction time interval. Each TS segment video file is extracted according to the corresponding frame extraction time interval, which can effectively reduce the amount of computation and maintain the integrity of the video content.

[0149] Then, cluster analysis is performed on the extracted video keyframes. To ensure that keyframes that express the main content of the video are extracted from multiple frames, irrelevant scene frames with low target occurrence frequency are filtered out. First, to preserve the temporal continuity between frames, temporal and spatial information is added to each target feature vector in the video keyframe. For example, the time and location of target feature vectors such as vehicles, people, and scenes contained in a certain video keyframe are added. The visual features, temporal and spatial information of the frames are considered during the cluster analysis. Then, unsupervised learning cluster analysis is performed on each corresponding target feature vector. By defining a cluster as the largest set of density-connected points (i.e., target feature vectors), high-density regions are divided into clusters, and each cluster can be found in the noisy data space. Target feature vectors with low occurrence frequency are considered to be discarded, which means that these target feature vectors do not belong to a cluster or cannot be constructed into a cluster during the clustering process. Target feature vectors with high occurrence frequency are considered to be retained, and then the first candidate keyframe set A after clustering is obtained. Through the above clustering analysis, video keyframes that only have target feature vectors that appear infrequently (i.e., irrelevant content scenes) can be filtered out, while video keyframes that have target feature vectors that appear infrequently can be retained, resulting in the first candidate keyframe set A after clustering.

[0150] Next, the video keyframes in the first candidate keyframe set A are converted from the RGB color space to the LUV color space, and the average difference intensity between consecutive frames is calculated. The specific calculation formula is as follows:

[0151]

[0152] Wherein d(I i I j ) represents video keyframe I i and video keyframes I j The average difference intensity between them, where m and n represent the number of pixels in the horizontal and vertical directions of the video keyframe, and p i (x, y) represents video keyframe I. i The pixel value at coordinates (x, y), p j (x, y) represents video keyframe I. j The pixel value at coordinate (x, y), i, j∈A.

[0153] For example, the video keyframes in the first candidate keyframe set A can be divided into equal groups, each group consisting of N consecutive video keyframes. The average difference intensity between M consecutive video keyframes can be calculated using the above formula (1).

[0154] For each group, the video keyframe with the largest average difference intensity is selected and added to the second candidate keyframe set Q. The first video keyframe from the second candidate keyframe set Q is then extracted and added to the final keyframe set R.

[0155] Iterate through the remaining video keyframes I in the second candidate keyframe set Q. a If any of the remaining video keyframes I in the second candidate keyframe set Q... a With any video keyframe I in the keyframe set R b If the distance is less than the inter-frame distance threshold t, then the video keyframe I is considered to be... a Redundant frames should be discarded; if the distance is greater than the inter-frame distance threshold t, then extract the video keyframe I. a Add it to the keyframe set R.

[0156] The inter-frame distance threshold t is determined based on the mean distance and standard deviation of distances between all video keyframes in the keyframe set R. For example, the inter-frame distance threshold t = mean distance μ + standard deviation δ between all video keyframes in the keyframe set R. By determining the inter-frame distance threshold using the mean distance and standard deviation of distances between all video keyframes in the keyframe set R, and dynamically changing with the number of video keyframes in the keyframe set R, the inter-frame distance threshold can be adaptively adjusted, improving the accuracy of effective keyframe extraction and the detection precision of keyframes.

[0157] Finally, the resulting keyframe set R consists of all valid video keyframes extracted from the video stream, i.e., view data, and is saved to the data storage module in image format.

[0158] Specifically, the structure enhancement processing of the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism yields the corresponding view structured data, including:

[0159] Save the extracted view data to the data storage module;

[0160] New view data is read from the data storage module in a polling manner according to a preset first time interval, and the new view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data.

[0161] The structured data of the view is saved to the data storage module.

[0162] The improved dilated spatial convolution pyramid pooling model with integrated attention mechanism includes a backbone network, an improved dilated spatial convolution pyramid pooling module, an attention mechanism module, a target feature extraction network, and a graph convolutional network.

[0163] Specifically, the step involves polling and reading newly added view data from the data storage module at a preset first time interval, and then performing structure enhancement processing on the newly added view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism to obtain the corresponding view structured data, including:

[0164] The backbone network is used to extract basic features from the newly added view data to obtain a first feature map representing high-level semantics.

[0165] The improved dilated spatial convolution pyramid pooling module is used to extract multi-scale information from the first feature map;

[0166] The multi-scale information is fused, and the fused multi-scale information is input into the attention mechanism module to obtain a scene semantic feature map;

[0167] The target feature map is obtained by extracting target features from the first feature map using the target feature extraction network.

[0168] The scene semantic feature map and the target feature map are input into the graph convolutional network to obtain semantic feature information;

[0169] The semantic feature information is processed in a structured manner to obtain the view structured data.

[0170] The structured data of the view includes label values, text feature vectors, and image feature vectors.

[0171] In this embodiment of the invention, the view data toolchain deployed on the algorithm service module can poll and read new view data from the data storage module at certain intervals; the view data toolchain of the algorithm service module can detect and analyze the new view data, and can perform enhanced view structuring processing through an improved dilated spatial convolution pyramid pooling model that integrates attention mechanisms.

[0172] The improved dilated spatial convolution pyramid pooling model that incorporates an attention mechanism consists of a backbone network N, an improved dilated spatial convolution pyramid pooling module P, an attention mechanism module R, and a graph convolutional network G.

[0173] The backbone network N can be a general backbone network built based on convolution or separable convolution, or a backbone network built based on separable dilated convolution, and is not specifically limited in the embodiments of the present invention.

[0174] The improved dilated spatial convolution pyramid pooling module P includes multiple convolution operations with different dilation rates. By using convolution operations with different dilation rates, the receptive field can be increased, multi-scale features can be captured, and the shortcomings of convolutional neural networks in processing complex features such as targets of different sizes, shapes and textures in images can be addressed.

[0175] The attention mechanism module R can focus on key features and suppress redundant information, thereby enhancing the network's ability to focus on targets and solving problems such as the difficulty in detection and recognition caused by the diverse appearance of targets in images and the presence of a lot of noise.

[0176] like Figure 5 As shown, the specific view structuring process is as follows:

[0177] First, the newly added view data I is input into the backbone network N, which consists of 7×7 convolutional layers, to extract basic features and obtain the first feature map FB with high-level semantics.

[0178] Then, the first feature map FB is resampled into the improved dilated spatial convolution pyramid pooling module P for multi-scale information extraction, realizing effective interaction between features at each level and fusing multi-scale information;

[0179] After cross-scale feature fusion, the multi-scale information input attention mechanism module R further improves the ability to perceive key information. It reduces the number of model parameters and extracts global features through global average pooling operation, and applies 1*1 convolutional layers to capture cross-channel interaction information, extract global relationship information, and obtain scene semantic feature map FR.

[0180] Simultaneously, the first feature map FB is input into the general target feature extraction network M to obtain the target feature map FM;

[0181] Finally, the scene semantic features (FR) and the target feature map (FM) are input into the graph convolutional network (G). Scene information is added to the target features to construct the relationship between the scene semantic feature map and the target feature map. The final semantic feature information is obtained through detection processing such as classification and regression. Based on the semantic feature information, the view data is structured to obtain its label values, feature values ​​(such as text features, image features, etc.), and other structured data. The processed view structured data is saved to the data storage module in the form of high-dimensional feature vectors.

[0182] By uploading and aggregating the structured view data in the data storage module to the cloud, a data foundation is provided for subsequent view big data processing and development.

[0183] In this embodiment of the invention, deploying a view data toolchain on edge processing nodes and data acquisition terminals can support automatic detection of video streaming protocols supported by video sources; using a frame difference method with fusion clustering analysis to efficiently and accurately extract key frames from the video, and completing the enhanced processing of view structuring through improved dilated spatial convolution pyramid pooling with fusion attention mechanism, providing a data foundation for the analysis and development of view big data, and meeting the needs of visual network application development and business operation.

[0184] The following section provides a detailed explanation of the view data processing methods involved in the above-mentioned big data aggregation and integration process.

[0185] Please see Figure 6 , Figure 6 This is another flowchart of a view data processing method provided in an embodiment of the present invention. The view data processing method, applied in the cloud, includes:

[0186] S21: Aggregate and store the view structured data from multiple data sources uploaded by the edge processing node; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism;

[0187] S22: The multi-modal pre-trained neural network model is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view to obtain the corresponding view fusion data;

[0188] S23: Classify and store the fused data of each view.

[0189] Furthermore, the method also includes:

[0190] Receive structured view data from multiple data sources uploaded by edge processing nodes according to business requirements;

[0191] The structured view data is obtained by the data acquisition terminal after performing structured processing on the view data it collects.

[0192] Alternatively, the structured view data is obtained by edge processing nodes performing structured processing on view data from multiple data sources.

[0193] It should be noted that the structured processing flow of the data acquisition end and edge processing nodes is as described above, and will not be repeated here.

[0194] This invention introduces cloud computing to build a leading big data capability and application system for the visual network. The structured view data from each edge processing node is distributed and stored, then uploaded and aggregated according to business needs. The data integration module allows for a one-time, full aggregation to the cloud. The cloud supports heterogeneous data source access, including databases and message queues. Supported databases include Oracle, MySQL, mysqlbinlog, and SQL Server; message queues include Kafka and RabbitMQ.

[0195] After receiving the structured view data uploaded by various edge processing nodes according to business needs, the cloud performs multi-dimensional heterogeneous data alignment on the heterogeneous structured view data based on a multimodal pre-trained neural network model. This enables the alignment of different data source attributes with data standards, providing a high-quality data foundation for the development, governance, service, and application of view data, and meeting the needs of visual network application development and business operation.

[0196] The multimodal pre-trained neural network model includes a view encoder, a contrastive learning module, and a multimodal fusion module.

[0197] Specifically, the step of aligning the structured data of each view with heterogeneous data using a multimodal pre-trained neural network model to obtain corresponding view fusion data includes:

[0198] For each of the view structured data, the view encoder extracts multimodal feature vectors from the view structured data to obtain the text features and image features of the corresponding view structured data;

[0199] The contrastive learning module performs semantic alignment between the text features and the image features to generate sequence features.

[0200] The sequence features are fused using the multimodal fusion module to obtain corresponding view fusion data; the view fusion data includes multimodal fusion feature values ​​obtained by deep fusion of the text features and the image features.

[0201] Furthermore, the method also includes:

[0202] By performing data standard mapping and alignment on the label values ​​of the same data attribute in the structured data of the views from different data sources, the standard label values ​​of the corresponding view fusion data are obtained.

[0203] Based on the standard label values ​​of the view fusion data, a query statement is generated for querying the view structured data of the same data attribute from multiple databases.

[0204] In this embodiment of the invention, the view encoder includes two single-modal extraction networks, a CNN convolutional neural network, and a Transformer encoder containing a feedforward neural network.

[0205] First, the structured view data is classified into text and image. The text and image are then fed into two single-modal extraction networks of the view encoder to extract text feature vectors and image feature vectors, respectively. The image feature vectors can be used to extract visual features through a CNN convolutional neural network to obtain the final image features. The text feature vectors can be used to process language information through a Transformer encoder containing a feedforward neural network to obtain the final text features.

[0206] Then, the two feature information are output to the contrastive learning module for processing. The model is trained by comparing positive samples (matching image-text pairs) and negative samples (mismatching image-text pairs). The similarity of the loss between the two types of samples is calculated to achieve semantic alignment of image features and text features and generate sequence features.

[0207] The initially aligned sequence features are then fed into the multimodal fusion module. By performing two pre-training tasks—image-text matching and masked language modeling—further fine-grained multimodal fusion is achieved, ultimately outputting multimodal feature values ​​after deep fusion of text and image features.

[0208] Finally, the label values ​​belonging to the same data attribute are standardized and transformed to align the attributes of different data sources with the data standard mapping. For example, for the attribute of vehicle, the label value of view structured data 1 is YY car, and the label value of view structured data 2 is XX car. By aligning with the data standard mapping, it can be converted into the standard label value: car.

[0209] The results of mapping and aligning the above data standards are engineered into database executable query statements, which can realize full heterogeneous data fusion query search, break the limitations of traditional view data transmission, and integrate with other related business data across domains to form a centralized, aligned, and efficient retrieval system. This can provide a high-quality data foundation for view data development and application, and meet the needs of visual network application development and business operation.

[0210] Specifically, the classification and storage of the fused view data includes:

[0211] The view fusion data is categorized and databases are built according to preset classification rules to obtain multiple databases.

[0212] For example, the merged view data can be classified and stored in databases according to the unified standards and process specifications of data definition. Based on the characteristics of the view data, databases such as the original database, theme database, resource database, knowledge base, business database, and business index database can be established for classified storage.

[0213] Furthermore, the method also includes:

[0214] Analyze the data application development tasks initiated by the user to determine the target data attributes;

[0215] The corresponding target view structured data is retrieved from the data storage module using the query statement corresponding to the target data attribute.

[0216] The target view structured data is mined and analyzed to obtain data application analysis results;

[0217] The data application analysis results are visualized according to the business requirements indicated by the data application development task to obtain the final data application analysis results.

[0218] The final data application analysis results are sent to the user terminal.

[0219] like Figure 3 As shown, after completing the above big data aggregation and fusion process, the big data application development process can be further executed. The specific process is as follows:

[0220] Users can initiate data application development tasks to the Visual Network Big Data Application (AS) through the platform's front-end page, based on their actual business needs.

[0221] The Visual Network Big Data Application (AS) parses the task content of the data application development task, including but not limited to business flow orchestration, data source, data type, data attributes, image or report presentation, and then sends the data application development task to the cloud.

[0222] The cloud-based data management module parses the task, determines the target data attributes (i.e., the required data types), and sends an upload data request to the classification and storage module.

[0223] The classification storage module generates corresponding query statements according to the target data attributes required in the uploaded data request, retrieves the required target view structured data from the database, and pushes the required target view structured data to the data calculation module according to the push rules and push cycle required in the uploaded data request.

[0224] The data computation module preprocesses the structured data of the target view, including but not limited to data cleaning, splicing, metadata mining, transformation and aggregation, and building access indexes.

[0225] The data analysis module performs data mining and analysis on the preprocessed target view structured data, such as using Madout, MLINB tools and other algorithm modeling tools to perform data mining and analysis, and returns the data application analysis results;

[0226] The data management module visualizes the data analysis structure, and the visualized data application analysis results are sent to the user terminal through the Visual Network Big Data Application (AS).

[0227] This invention focuses on view data, encompassing the entire process from video stream acquisition, structured processing, uploading and aggregation, cloud-based data integration, multi-source heterogeneous data alignment, storage, computation, management, and application. By incorporating inter-frame difference algorithms with clustering analysis to extract keyframes from raw data acquired by traditional acquisition devices and new computing power devices, the detection accuracy of video keyframes can be improved. Structured enhancement processing of the raw view data extracted using an improved dilated spatial convolution pyramid pooling model with an attention mechanism enables efficient generation of structured data. Furthermore, cloud-based alignment of multi-source heterogeneous view structured data using a multimodal pre-trained neural network model enables view structured data fusion, providing a high-quality data foundation for view data development and applications. This supports richer view data-related innovative capabilities and applications beyond view data recognition and sharing, meeting the needs of visual network application development and business operations.

[0228] See Figure 7 , Figure 7 This is a structural block diagram of a view data processing device provided in an embodiment of the present invention. The view data processing device is applied to an edge processing node or a data acquisition terminal and includes: a video access module 11, an algorithm service module 12, and a data upload module 13.

[0229] The video access module 11 is used to pull video streams from the data source;

[0230] The algorithm service module 12 is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism, so as to obtain the view structured data of the corresponding view data.

[0231] The data upload module 13 is used to upload the view structured data to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

[0232] In an optional embodiment, the algorithm service module 12 includes:

[0233] The first data storage unit is used to save the extracted view data to the data storage module;

[0234] The structured processing unit is used to poll and read newly added view data from the data storage module at a preset first time interval, and to perform structured enhancement processing on the newly added view data using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data.

[0235] The second data storage unit is used to save the view structured data to the data storage module.

[0236] In one optional embodiment, the video access module 11 includes:

[0237] Protocol detection unit, used to poll and detect the video streaming protocols supported by the data source;

[0238] The video retrieval unit is used to retrieve a video stream from the data source according to the video streaming protocol supported by the data source.

[0239] In an optional embodiment, the video access module 11 further includes:

[0240] The inter-frame difference calculation unit is used to extract keyframes from the video stream using an inter-frame difference algorithm based on fusion clustering analysis to obtain view data of the video stream.

[0241] In one optional embodiment, the inter-frame difference calculation unit includes:

[0242] The keyframe extraction subunit is used to extract video keyframes from the video stream according to the frame extraction time interval.

[0243] The clustering analysis subunit is used to perform clustering analysis on the extracted video keyframes to filter scene frames in the video keyframes and obtain a first candidate keyframe set.

[0244] The inter-frame difference processing subunit is used to perform inter-frame difference processing on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream.

[0245] In one optional embodiment, the keyframe extraction subunit includes:

[0246] The TS slicing subunit is used to slice the video stream into TS segments according to a preset second time interval to obtain multiple TS slice video files.

[0247] The frame extraction time interval determination subunit is used to determine the frame extraction time interval of the corresponding TS slice video file based on the duration of the TS slice video file and the video frame rate of the corresponding TS slice video file.

[0248] The first frame extraction subunit is used to extract video keyframes from each of the TS slice video files according to the frame extraction time interval.

[0249] In one optional embodiment, the inter-frame differential processing subunit includes:

[0250] The color space conversion subunit is used to convert the video keyframes in the first candidate keyframe set from RGB color space to LUV color space.

[0251] The average difference intensity calculation subunit is used to calculate the average difference intensity between several consecutive video keyframes in the first candidate keyframe set in the LUV color space, and extract the video keyframe with the largest average difference intensity among several consecutive video keyframes and save it to the second candidate keyframe set.

[0252] The second frame extraction subunit is used to extract the first video keyframe from the second candidate keyframe set into the keyframe set.

[0253] The judgment subunit is used to determine whether the distance between the next video key frame in the second candidate key frame set and any video key frame in the key frame set is less than a preset inter-frame distance threshold.

[0254] The frame discard sub-unit is used to discard the next video keyframe if necessary.

[0255] The third frame extraction subunit is used to extract the next video keyframe to the keyframe set if no.

[0256] The video keyframes in the keyframe set are view data from the corresponding data source.

[0257] In one optional embodiment, the improved dilated spatial convolution pyramid pooling model with integrated attention mechanism includes a backbone network, an improved dilated spatial convolution pyramid pooling module, an attention mechanism module, a target feature extraction network, and a graph convolutional network.

[0258] The structured processing unit includes a data structuring subunit, which performs the following processes:

[0259] The backbone network is used to extract basic features from the newly added view data to obtain a first feature map representing high-level semantics.

[0260] The improved dilated spatial convolution pyramid pooling module is used to extract multi-scale information from the first feature map;

[0261] The multi-scale information is fused, and the fused multi-scale information is input into the attention mechanism module to obtain a scene semantic feature map;

[0262] The target feature map is obtained by extracting target features from the first feature map using the target feature extraction network.

[0263] The scene semantic feature map and the target feature map are input into the graph convolutional network to obtain semantic feature information;

[0264] The semantic feature information is processed in a structured manner to obtain the view structured data.

[0265] It should be noted that the working process of each module in the view data processing device described in the embodiments of the present invention can refer to the working process of the view data processing method described in the above embodiments, and the technical effect achieved is the same as that of the view data processing method described in the above embodiments, so it will not be repeated here.

[0266] See Figure 8 , Figure 8 This is another structural block diagram of a view data processing device provided in an embodiment of the present invention. The view data processing device is applied to the cloud and includes: a data integration module 21, a data calculation module 22, and a classification storage module 23.

[0267] The data integration module 21 is used to aggregate and store the view structured data from multiple data sources uploaded by the edge processing node; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism;

[0268] The data calculation module 22 is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data;

[0269] The classification storage module 23 is used to classify and store the fused data of each view.

[0270] In one optional embodiment, the multimodal pre-trained neural network model includes a view encoder, a contrastive learning module, and a multimodal fusion module;

[0271] The data calculation module 22 includes a data fusion unit, which performs the following process:

[0272] For each of the view structured data, the view encoder extracts multimodal feature vectors from the view structured data to obtain the text features and image features of the corresponding view structured data;

[0273] The contrastive learning module performs semantic alignment between the text features and the image features to generate sequence features.

[0274] The sequence features are fused using the multimodal fusion module to obtain corresponding view fusion data; the view fusion data includes multimodal fusion feature values ​​obtained by deep fusion of the text features and the image features.

[0275] In an optional embodiment, the classification storage module 23 includes:

[0276] The classification and database building unit is used to classify and build multiple databases by the view fusion data according to preset classification rules.

[0277] As an improvement to the above solution, the data calculation module 22 includes:

[0278] The data standard mapping and alignment unit is used to perform data standard mapping and alignment on the label values ​​of the same data attribute in the view structured data from different data sources to obtain the standard label values ​​of the corresponding view fusion data.

[0279] The query statement generation unit is used to generate a query statement for searching the same data attribute of the view structured data from multiple databases, based on the standard label values ​​of the view fusion data.

[0280] In one alternative embodiment, the device includes:

[0281] The data receiving module is used to receive view structured data from multiple data sources uploaded by edge processing nodes according to business requirements;

[0282] The structured view data is obtained by the data acquisition terminal after performing structured processing on the view data it collects.

[0283] Alternatively, the structured view data is obtained by edge processing nodes performing structured processing on view data from multiple data sources.

[0284] It should be noted that the working process of each module in the view data processing device described in the embodiments of the present invention can refer to the working process of the view data processing method described in the above embodiments, and the technical effect achieved is the same as that of the view data processing method described in the above embodiments, so it will not be repeated here.

[0285] See Figure 9 , Figure 9 This invention provides a structural block diagram of a view data processing system, which includes: an edge processing node 100, a cloud 200, and a data acquisition terminal 300; the edge processing node 100 includes: a video access module 11, an algorithm service module 12, and a data upload module 13; the cloud 200 includes: a data integration module 21, a data calculation module 22, and a classification storage module 23.

[0286] The video access module 11 is used to pull video streams from the data sources corresponding to each data acquisition terminal and extract view data of the video streams;

[0287] The algorithm service module 12 is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism, so as to obtain the view structured data of the corresponding view data.

[0288] The data upload module 13 is used to upload the view structured data to the cloud through the edge processing node;

[0289] The data integration module 21 is used to aggregate and store the view structured data from multiple data sources uploaded by edge nodes.

[0290] The data calculation module 22 is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data;

[0291] The classification storage module 23 is used to classify and store the fused data of each view.

[0292] It should be noted that the working process of each module in the view data processing system described in the embodiments of the present invention can refer to the working process of the view data processing method described in the above embodiments, and the technical effect achieved is the same as that of the view data processing method described in the above embodiments, so it will not be repeated here.

[0293] See Figure 10 , Figure 10 This is a structural block diagram of a view data processing device provided in an embodiment of the present invention. The view data processing device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described view data processing method embodiments, such as steps S11 to S13 or steps 21 to S23.

[0294] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the view data processing device.

[0295] The view data processing device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of a view data processing device and does not constitute a limitation on the view data processing device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the view data processing device may also include input / output devices, network access devices, buses, etc.

[0296] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the view data processing device, connecting all parts of the view data processing device via various interfaces and lines.

[0297] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the view data processing device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0298] If the modules / units integrated into the view data processing device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0299] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0300] The above description is a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, many improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A view data processing method, characterized in that, Applied to edge processing nodes or data acquisition terminals, including: Pull the video stream from the data source and extract the view data of the video stream; The view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structure data of the corresponding view data. The view structured data is uploaded to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

2. The view data processing method as described in claim 1, characterized in that, The view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism to obtain the corresponding view structured data, including: Save the extracted view data to the data storage module; New view data is read from the data storage module in a polling manner according to a preset first time interval, and the new view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data. The structured data of the view is saved to the data storage module.

3. The view data processing method as described in claim 1, characterized in that, The process of pulling the video stream from the data source includes: Poll the data source to detect the video streaming protocols it supports; The video stream is retrieved from the data source according to the video streaming protocol supported by the data source.

4. The view data processing method as described in claim 1, characterized in that, The extraction of view data from the video stream includes: The video stream is subjected to keyframe extraction using an inter-frame difference algorithm based on fusion clustering analysis to obtain the view data of the video stream.

5. The view data processing method as described in claim 4, characterized in that, The inter-frame difference algorithm using fusion clustering analysis is used to extract keyframes from the video stream to obtain view data of the video stream, including: Video keyframes are extracted from the video stream according to the frame extraction time interval. Cluster analysis is performed on the extracted video keyframes to filter out scene frames in the video keyframes, and a first candidate keyframe set is obtained. Inter-frame difference processing is performed on the video keyframes in the first candidate keyframe set to obtain the view data of the video stream.

6. The view data processing method as described in claim 5, characterized in that, The step of extracting keyframes from the video stream according to the frame extraction time interval includes: The video stream is sliced ​​into TS segments according to a preset second time interval to obtain multiple TS segment video files; The frame extraction time interval of the corresponding TS segment video file is determined based on the duration of the TS segment video file and the video frame rate of the corresponding TS segment video file. Video keyframes are extracted from each of the TS slice video files according to the frame extraction time interval.

7. The view data processing method as described in claim 5, characterized in that, The step of performing inter-frame difference processing on video keyframes in the first candidate keyframe set to obtain view data of the video stream includes: Convert the video keyframes in the first candidate keyframe set from the RGB color space to the LUV color space; In the LUV color space, the average difference intensity between several consecutive video keyframes in the first candidate keyframe set is calculated, and the video keyframe with the largest average difference intensity among several consecutive video keyframes is extracted and saved to the second candidate keyframe set. Extract the first video keyframe from the second candidate keyframe set into the keyframe set; Determine whether the distance between the next video keyframe in the second candidate keyframe set and any video keyframe in the keyframe set is less than a preset inter-frame distance threshold. If so, discard the next video keyframe; If not, extract the next video keyframe to the keyframe set; The video keyframes in the keyframe set are view data from the corresponding data source.

8. The view data processing method as described in claim 2, characterized in that, The improved dilated spatial convolution pyramid pooling model with integrated attention mechanism includes a backbone network, an improved dilated spatial convolution pyramid pooling module, an attention mechanism module, a target feature extraction network, and a graph convolution network. The process involves polling and reading newly added view data from the data storage module at preset first time intervals, and then performing structure enhancement processing on the newly added view data using an improved dilated spatial convolution pyramid pooling model with fused attention mechanism to obtain the corresponding view structured data, including: The backbone network is used to extract basic features from the newly added view data to obtain a first feature map representing high-level semantics. The improved dilated spatial convolution pyramid pooling module is used to extract multi-scale information from the first feature map; The multi-scale information is fused, and the fused multi-scale information is input into the attention mechanism module to obtain a scene semantic feature map; The target feature map is obtained by extracting target features from the first feature map using the target feature extraction network. The scene semantic feature map and the target feature map are input into the graph convolutional network to obtain semantic feature information; The semantic feature information is processed in a structured manner to obtain the view structured data.

9. A method for processing view data, characterized in that, Applied to the cloud, including: The view structured data from multiple data sources uploaded by edge processing nodes is aggregated and stored; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism; By using a multimodal pre-trained neural network model to align the structured data of each view with heterogeneous data, corresponding view fusion data is obtained. The data from each of the aforementioned view fusions is categorized and stored.

10. The view data processing method as described in claim 9, characterized in that, The multimodal pre-trained neural network model includes a view encoder, a contrastive learning module, and a multimodal fusion module; The step of aligning the structured data of each view with heterogeneous data using a multimodal pre-trained neural network model to obtain corresponding view fusion data includes: For each of the view structured data, the view encoder extracts multimodal feature vectors from the view structured data to obtain the text features and image features of the corresponding view structured data; The contrastive learning module performs semantic alignment between the text features and the image features to generate sequence features. The sequence features are fused using the multimodal fusion module to obtain corresponding view fusion data; the view fusion data includes multimodal fusion feature values ​​obtained by deep fusion of the text features and the image features.

11. The view data processing method as described in claim 9, characterized in that, The process of classifying and storing the fused view data includes: The view fusion data is categorized and databases are built according to preset classification rules to obtain multiple databases.

12. The view data processing method as described in claim 11, characterized in that, The method further includes: By performing data standard mapping and alignment on the label values ​​of the same data attribute in the structured data of the views from different data sources, the standard label values ​​of the corresponding view fusion data are obtained. Based on the standard label values ​​of the view fusion data, a query statement is generated for querying the view structured data of the same data attribute from multiple databases.

13. The view data processing method as described in claim 9, characterized in that, The method further includes: Receive structured view data from multiple data sources uploaded by edge processing nodes according to business requirements; The structured view data is obtained by the data acquisition terminal after performing structured processing on the view data it collects. Alternatively, the structured view data is obtained by edge processing nodes performing structured processing on view data from multiple data sources.

14. The view data processing method as described in claim 13, characterized in that, The structured processing of the view data includes: Pull the video stream from the data source; Extract the view data of the video stream and save the extracted view data to the data storage module; New view data is read from the data storage module in a polling manner according to a preset first time interval, and the new view data is subjected to structure enhancement processing using an improved dilated spatial convolution pyramid pooling model with fusion attention mechanism to obtain the view structured data of the corresponding view data.

15. A view data processing apparatus, characterized in that, Applied to edge processing nodes or data acquisition terminals, including: video access module, algorithm service module, and data upload module; The video access module is used to pull video streams from the data source; The algorithm service module is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism, so as to obtain the view structured data of the corresponding view data. The data upload module is used to upload the view structured data to the cloud through the edge processing node, so that the cloud can perform multi-modal heterogeneous data alignment on each view structured data through a multimodal pre-trained neural network model to obtain corresponding view fusion data.

16. A view data processing apparatus, characterized in that, Applied to the cloud, it includes: a data integration module, a data computing module, and a categorized storage module; The data integration module is used to aggregate and store view structured data from multiple data sources uploaded by edge processing nodes; wherein, the view structured data is obtained by performing structure enhancement processing on the view data through an improved dilated spatial convolution pyramid pooling model with fused attention mechanism; The data calculation module is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data. The classification and storage module is used to classify and store the fused data of each view.

17. A view data processing system, characterized in that, include: Cloud, edge processing nodes, and data acquisition terminals; The edge processing node includes: a video access module, an algorithm service module, and a data upload module; the cloud includes: a data integration module, a data computing module, and a categorized storage module. The video access module is used to pull video streams from the data sources corresponding to each data acquisition terminal and extract view data of the video streams; The algorithm service module is used to perform structure enhancement processing on the view data using an improved dilated spatial convolution pyramid pooling model with a fusion attention mechanism, so as to obtain the view structured data of the corresponding view data. The data upload module is used to upload the view structured data to the cloud through the edge processing node; The data integration module is used to aggregate and store the view structured data from multiple data sources uploaded by edge nodes. The data calculation module is used to perform multi-dimensional heterogeneous data alignment on the structured data of each view through a multimodal pre-trained neural network model to obtain corresponding view fusion data. The classification and storage module is used to classify and store the fused data of each view.

18. A view data processing device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the view data processing method as described in any one of claims 1 to 8 or the view data processing method as described in any one of claims 9 to 14.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the view data processing method as described in any one of claims 1 to 8 or the view data processing method as described in any one of claims 9 to 14.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the view data processing method according to any one of claims 1 to 8 or the view data processing method according to any one of claims 9 to 14.

Images