Video convergence management system based on large language model

Abstract

The invention discloses a video convergence management system based on a large-scale language model, relates to the technical field of video management, and solves the technical problems that a current monitoring video cannot cover a panorama and is restricted by a data transmission rate, and a video data set is difficult to efficiently train a video large-scale model. Comprising a video splicing module used for splicing and fusing received video streams to obtain panoramic video data; the cloud server is used for visually displaying the panoramic video data through a GIS geographic information system to realize real-time map presentation of a workshop production scene; the model training module adopts different data loading modes for data sets of different scales, and then trains the data sets to obtain an abnormal behavior detection model, so that the model training efficiency is maximized; and the safety analysis module is used for carrying out abnormal sign identification on the panoramic video data by applying an abnormal sign detection model so as to evaluate the safety condition of a workshop production scene, and the video analysis efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video management, in particular to a video aggregation management system based on a large language model. Background Art

[0002] With the rapid development of computer vision, artificial intelligence, communication technology, processor hardware and other application technologies, video surveillance equipment, with its huge advantages such as rich information, multiple types of supported tasks and low cost, is playing an increasingly important role in production, life and urban management. At present, the deployment and application of various video equipment are becoming more and more extensive, and the amount of visual data is also increasing sharply. How to effectively and fully use and process video big data is an important issue related to the development of various application fields;

[0003] Due to the dispersion of network security and video surveillance areas, surveillance videos from different channels will have overlapping parts, resulting in poor visual sensory effects, and surveillance videos from different channels may have gaps, failing to cover the full view, and the overlapping video data will waste storage space; at the same time, since the scale of video data sets is still growing, it is difficult to use video big data to efficiently train video large models due to the constraints of data transmission rate, resulting in restricted video management efficiency; based on the above shortcomings, the present invention proposes a video aggregation management system based on a large language model. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a video aggregation management system based on a large language model.

[0005] To achieve the above-mentioned object, a first aspect of the present invention provides a video aggregation management system based on a large language model, comprising a video acquisition module, an image preprocessing module, a video stitching module, a cloud server, a security analysis module and a model training module;

[0006] The video acquisition module is used to acquire a multi-view image sequence of a workshop production scene through a plurality of fixed-focus lenses, and transmit the acquired multi-view image sequence to the image preprocessing module;

[0007] The image preprocessing module is used to preprocess each frame of video image in the multi-view image sequence to enhance image details, contrast and clarity; the specific processing steps are as follows:

[0008] Obtain each frame of video image in the multi-view image sequence, filter, sharpen, perform mathematical morphological transformation, binarize, extract edges and extract contours on the single frame of video image to obtain pre-processed image data;

[0009] Then, the preprocessed image data is subjected to brightness detection to obtain a brightness deviation value Lp, and the brightness of the preprocessed image data is adjusted to a preset brightness threshold value based on the brightness deviation value Lp;

[0010] Finally, the adjusted pre-processed image data is converted into a digital signal and compressed and encoded to form a video stream; the video stream is embedded with unified time code information;

[0011] The image preprocessing module is used to transmit the video stream to the video stitching module via Ethernet for stitching and fusion to obtain panoramic video data; the video stitching module is used to feed back the panoramic video stream to the cloud server;

[0012] The cloud server is used to visualize the panoramic video data through the GIS geographic information system to achieve real-time mapping of workshop production scenes;

[0013] The safety analysis module is used to apply an abnormal sign detection model to identify abnormal signs in panoramic video data to evaluate the safety of workshop production scenes; the abnormal sign detection model is established based on the Bi-LSTM model of the Attention mechanism and is trained using historical video data of workshop production scenes.

[0014] Furthermore, the specific analysis steps of the security analysis module include:

[0015] Obtain each frame of video image data in the panoramic video data, crop and segment the single frame of video image data to obtain multiple scene images; input each scene image into the abnormal sign detection model, identify the abnormal signs in the scene image, and classify the abnormal signs;

[0016] The abnormal signs are designed to be broken glass or rivets scattered on the ground, the stopped working state of the machine, other improperly placed equipment or materials, and incorrect operating behaviors in various production processes;

[0017] The cloud server is used to drive the early warning module to issue an early warning prompt according to the abnormal sign type; each abnormal sign type corresponds to a prompt type.

[0018] Furthermore, the model training module is used to load the data set used for training, establish a Bi-LSTM model based on the Attention mechanism, so as to train the data set to obtain an abnormal behavior detection model; the data set is historical video data of the workshop production scene.

[0019] Furthermore, the model training module includes a number of training nodes and a network switch;

[0020] The training node is used to obtain training and verification data from local memory or from the cloud server through the network switch to complete specific model training tasks; the network switch is used to connect multiple training nodes and cloud servers to provide direct address mapping and high-speed data transmission channels for data access.

[0021] Furthermore, the model training module has two data loading methods for loading the training data set, namely, memory loading method and cloud server loading method; the specific loading steps are as follows:

[0022] Obtain the storage space occupied by the training data set and mark it as N1; obtain the remaining memory size of the training node and mark it as N2; calculate the memory share Nb using the formula Nb=N1 / N2;

[0023] Compare the memory usage Nb with the preset threshold; if Nb < the preset threshold, it means that the dataset is not large, and the dataset is directly read through memory loading; if the memory usage Nb ≥ the preset threshold, it means that the dataset is large, and the dataset is read concurrently through cloud server loading.

[0024] Furthermore, the memory loading method refers to: allocating a space from the training node memory as a dedicated storage space, and loading the training data set into the dedicated storage space in advance before the training starts, so as to be directly read during training.

[0025] Furthermore, the cloud server loading method refers to: distributing and storing the data set in the cloud server, and for each iteration cycle of the training process, multiple training nodes concurrently read corresponding batches of data from the cloud server through the network switch, and temporarily store them in the respective memories of the multiple training nodes for training use.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The image preprocessing module in the present invention is used to preprocess each frame of video image in the multi-view image sequence to enhance image details, contrast and clarity, and provide solid data support for subsequent video analysis; then the adjusted image data is converted into digital signals and compressed and encoded to form a video stream; the video splicing module is used to splice and fuse the received video streams to obtain panoramic video data; the cloud server is used to visualize the panoramic video data through the GIS geographic information system to realize the real-time mapping presentation of the workshop production scene, which is convenient for management personnel to understand in all aspects;

[0028] 2. The model training module in the present invention is used to load the data set used for training, establish a Bi-LSTM model based on the Attention mechanism to train the data set, and obtain an abnormal behavior detection model; the safety analysis module is used to apply the abnormal sign detection model to identify abnormal signs of panoramic video data, so as to evaluate the safety of workshop production scenes and effectively improve the efficiency of video analysis; at the same time, different data loading methods are adopted for data sets of different sizes to maximize the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 The present invention is a system block diagram of a video aggregation management system based on a large language model.

[0031] Figure 2 It is a structural diagram of the model training module in the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 to Figure 2 , the first aspect of the present invention provides a video aggregation management system based on a large language model, including a video acquisition module, an image preprocessing module, a video stitching module, a cloud server, a security analysis module, an early warning module and a model training module;

[0034] The video acquisition module is used to obtain a multi-view image sequence of the workshop production scene through a number of fixed-focus lenses; the multi-view image sequence has no less than 5 viewing angles, namely, front view, rear view, left view, right view and bottom view, forming a 360-degree coverage; among which, the image resolution taken by the fixed-focus lens is not less than 5cm;

[0035] Specifically, a number of fixed-focus lenses are arranged in a fan-shaped arrangement, and the videos shot by two adjacent fixed-focus lenses partially overlap; since the fixed-focus lenses always maintain basically the same horizontal plane and their positions are relatively fixed, the image quality captured by the video acquisition module is high, and there will be no unfavorable video splicing situations such as rotation and not being in the same horizontal plane, effectively ensuring the fineness of the video splicing effect;

[0036] The video acquisition module is used to transmit the acquired multi-view image sequence to the image preprocessing module; the image preprocessing module is used to preprocess each frame of the video image in the multi-view image sequence to enhance the image details, contrast and clarity, and provide solid data support for subsequent video analysis;

[0037] In this embodiment, the specific processing steps of the image preprocessing module are as follows:

[0038] Obtain each frame of video image in the multi-view image sequence, filter, sharpen, perform mathematical morphological transformation, binarize, extract edges and extract contours on the single frame of video image to obtain pre-processed image data;

[0039] Then, the brightness of the pre-processed image data is detected, and the brightness of the pre-processed image data is adjusted to a preset brightness threshold based on the brightness deviation value Lp; specifically:

[0040] Compare the detection result with the preset brightness threshold to obtain the brightness deviation value and mark it as Lp;

[0041] For example, if the brightness deviation value Lp is a positive number, it indicates that the brightness of the pre-processed image data is higher than the preset brightness threshold, and the brightness needs to be lowered; if the brightness deviation value Lp is a negative number, it indicates that the brightness of the pre-processed image data is lower than the preset brightness threshold, and the brightness needs to be increased; if the brightness deviation value Lp is 0, it indicates that there is no need to adjust the brightness;

[0042] Finally, the adjusted pre-processed image data is converted into digital signals and compressed and encoded to form a video stream; the video stream is embedded with unified time code information;

[0043] The image preprocessing module is used to transmit the video stream to the video splicing module via Ethernet;

[0044] The video stitching module is used to stitch and fuse the received video streams to obtain panoramic video data, and feed the panoramic video streams back to the cloud server;

[0045] The cloud server is used to visualize the panoramic video data through the GIS geographic information system, realizing the real-time mapping presentation of the workshop production scene, which is convenient for managers to understand in all aspects;

[0046] In this embodiment, the safety analysis module is used to apply an abnormal sign detection model to identify abnormal signs in the panoramic video data, so as to assess the safety of the workshop production scene; specifically, it includes:

[0047] Obtain each frame of video image data in the panoramic video data, crop and segment the single frame of video image data to obtain multiple scene images; input each scene image into the abnormal sign detection model, identify the abnormal signs in the scene image, and classify the abnormal signs;

[0048] For example, in a workshop production scenario, if there is broken glass or rivets scattered on the ground, a machine is stopped, or other improperly placed equipment or materials within the activity range, there will be safety hazards.

[0049] And in the production process, if the operator makes an incorrect operation, there will be a safety hazard;

[0050] Therefore, in this embodiment, the abnormal signs are designed to be broken glass or rivets scattered on the ground, the stopped working state of the machine, other improperly placed equipment or materials, and incorrect operating behaviors in various production processes;

[0051] The cloud server is used to drive the early warning module to issue early warning prompts according to the abnormal sign type to maintain production safety; each abnormal sign type corresponds to a prompt type;

[0052] In this embodiment, the model training module is used to load the data set used for training, establish a Bi-LSTM model based on the Attention mechanism to train the data set, and obtain an abnormal behavior detection model; the data set is historical video data of the workshop production scene;

[0053] Specifically, the model training module includes several training nodes and network switches;

[0054] The training node is used to obtain training and verification data from local memory or from the cloud server through a network switch to complete specific model training tasks;

[0055] The network switch is used to connect multiple training nodes and cloud servers, providing direct address mapping and high-speed data transmission channels for data access; the cloud server is a distributed storage server;

[0056] A further technical solution is that: the present application adopts different data loading methods for data sets of different sizes to maximize the model training efficiency; there are two data loading methods for loading the data sets used for training; the first is memory loading, and the second is cloud server loading;

[0057] The specific steps for the model training module to load the training data set are as follows:

[0058] Get the storage space occupied by the training data set and mark it as N1; get the remaining memory size of the training node and mark it as N2; use the formula Nb = N1 / N2 to calculate the memory share Nb;

[0059] Compare the memory share Nb with the preset threshold; if Nb < the preset threshold, it means that the data set is not large, and a space is allocated from the training node memory as a dedicated storage space; before the training starts, load the training data set into the dedicated storage space in advance for direct reading during training;

[0060] This application directly reads the data set through memory loading when the data set size is not large, which can greatly reduce the data loading time and greatly improve the training speed;

[0061] It should be understood that when the dataset is large, if the dataset is directly read through memory loading, the data transmission rate will be restricted, which will greatly affect the model training speed;

[0062] If the memory share Nb ≥ the preset threshold, it means that the data set is large in size. In this case, the data set is distributed and stored in the cloud server. For each iteration cycle of the training process, multiple training nodes concurrently read the corresponding batch of data from the cloud server through the network switch and temporarily store it in the memory of each training node for training.

[0063] When the data set is large, the application concurrently reads the corresponding batches of data through cloud server loading, which can greatly improve the data concurrency performance and transmission rate, and solve the problem of large-scale data storage and parallel fast loading in model training.

[0064] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0065] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A video aggregation management system based on a large language model, characterized in that: It includes video acquisition module, image preprocessing module, video stitching module, security analysis module and model training module; The video acquisition module is used to acquire a multi-view image sequence of a workshop production scene through a plurality of fixed-focus lenses, and transmit the acquired multi-view image sequence to the image preprocessing module; The image preprocessing module is used to preprocess each frame of video image in the multi-view image sequence to enhance image details, contrast and clarity; the specific processing steps are as follows: Obtain each frame of video image in the multi-view image sequence, filter, sharpen, perform mathematical morphological transformation, binarize, extract edges and extract contours on the single frame of video image to obtain pre-processed image data; Then, the preprocessed image data is subjected to brightness detection to obtain a brightness deviation value Lp, and the brightness of the preprocessed image data is adjusted to a preset brightness threshold value based on the brightness deviation value Lp; Finally, the adjusted pre-processed image data is converted into a digital signal and compressed and encoded to form a video stream; the video stream is embedded with unified time code information; The image preprocessing module is used to transmit the video stream to the video splicing module via Ethernet for splicing and fusion to obtain panoramic video data; The video stitching module is used to feed the panoramic video stream back to the cloud server; The cloud server is used to visualize the panoramic video data through the GIS geographic information system to achieve real-time mapping of workshop production scenes; The safety analysis module is used to apply an abnormal sign detection model to identify abnormal signs in panoramic video data to evaluate the safety of workshop production scenes; the abnormal sign detection model is established based on the Bi-LSTM model of the Attention mechanism and is trained using historical video data of workshop production scenes.

2. A video convergence management system based on a large language model according to claim 1, characterized in that: The specific analysis steps of the security analysis module include: Obtain each frame of video image data in the panoramic video data, crop and segment the single frame of video image data to obtain multiple scene images; input each scene image into the abnormal sign detection model, identify the abnormal signs in the scene image, and classify the abnormal signs; The abnormal signs are designed to be broken glass or rivets scattered on the ground, the stopped working state of the machine, other improperly placed equipment or materials, and incorrect operating behaviors in various production processes; The cloud server is used to drive the early warning module to issue an early warning prompt according to the abnormal sign type; each abnormal sign type corresponds to a prompt type.

3. The video aggregation management system based on a large language model according to claim 1, characterized in that: The model training module is used to load the data set used for training, establish a Bi-LSTM model based on the Attention mechanism, so as to train the data set and obtain an abnormal behavior detection model; the data set is historical video data of workshop production scenes.

4. The video convergence management system based on a large language model according to claim 3, characterized in that: The model training module includes several training nodes and network switches; The training node is used to obtain training and verification data from local memory or from the cloud server through the network switch to complete specific model training tasks; the network switch is used to connect multiple training nodes and cloud servers to provide direct address mapping and high-speed data transmission channels for data access.

5. The video aggregation management system based on a large language model according to claim 4, characterized in that: The model training module loads the training data set in two ways: memory loading and cloud server loading. The specific loading steps are as follows: Obtain the storage space occupied by the training data set and mark it as N1; obtain the remaining memory size of the training node and mark it as N2; calculate the memory share Nb using the formula Nb=N1 / N2; Compare the memory share Nb with a preset threshold; If Nb is less than the preset threshold, it means that the dataset is not large, and the dataset is directly read through memory loading. If the memory share Nb is greater than or equal to the preset threshold, it means that the dataset is large, and the dataset is read concurrently through cloud server loading.

6. The video convergence management system based on a large language model according to claim 5, characterized in that: The memory loading method refers to: allocating a space from the training node memory as a dedicated storage space, and loading the training data set into the dedicated storage space before the training starts, so that it can be directly read during training.

7. The video aggregation management system based on a large language model according to claim 5, characterized in that: The cloud server loading method refers to: distributing and storing the data set in the cloud server. For each iteration cycle of the training process, multiple training nodes concurrently read corresponding batches of data from the cloud server through the network switch, and temporarily store them in the respective memories of the multiple training nodes for training use.

Images