A video image structuring method and system based on work site

Abstract

The invention relates to the technical field of image data processing, and discloses a video image structuring method and system based on a work site. The method obtains a video frame sequence by deframing a monitoring video, extracts key frames from the video frame sequence by a histogram comparison method, extracts a 128-dimensional SIFT feature vector corresponding to a video image of each key frame based on a SIFT algorithm, and normalizes the 128-dimensional SIFT feature vector. A classifier model is constructed, the 128-dimensional SIFT feature vector is input into the classifier model, a corresponding semantic category label is output, and the semantic category label is added to a corresponding key frame, thereby obtaining a monitoring video containing a video label, so as to establish a semantic structure of video data, and improve the efficiency and accuracy of interpreting video information.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a video image structuring method and system based on a work site. Background Art

[0002] As an important part of on-site safety monitoring of power operations, video surveillance systems face huge challenges in deep application. The current problems that need to be solved are efficient extraction of video information, standard data exchange and semantic interoperability between various information systems.

[0003] However, video data is currently still browsed and checked manually. At the same time, there is a lack of unified structured standards and specifications, which makes it difficult to interpret video information efficiently and correctly. Summary of the invention

[0004] The present invention provides a video image structuring method and system based on a work site, which are used to solve the technical problem of low efficiency and accuracy in interpreting video information.

[0005] In view of this, a first aspect of the present invention provides a method for structuring a video image based on a work site, comprising the following steps:

[0006] Obtaining a surveillance video of the work site, de-framing the surveillance video, and obtaining a video frame sequence;

[0007] The similarity between two adjacent video frames in the video frame sequence is calculated by using a histogram comparison method, a video frame whose similarity is less than a preset threshold is extracted as a key frame, and the key frame is added to the key frame sequence;

[0008] Extracting underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm to obtain a 128-dimensional SIFT feature vector corresponding to the key frame of each frame;

[0009] Normalizing the 128-dimensional SIFT feature vector to obtain a normalized feature vector;

[0010] Based on the discriminative random field model, the previously acquired historical work site video data is trained, and the pre-set semantic category labels are used as output to build a classifier model;

[0011] Inputting the normalized feature vector into the classifier model, thereby outputting a semantic category label, and obtaining a mapping relationship between the normalized feature vector and the semantic category label;

[0012] The corresponding key frames are marked with video labels according to the mapping relationship between the normalized feature vector and the semantic category label, so as to obtain a monitoring video containing the video label.

[0013] Preferably, the step of using a histogram comparison method to calculate the similarity between two adjacent video frames in the video frame sequence, extracting video frames whose similarity is less than a preset threshold as key frames, and adding the key frames to the key frame sequence specifically includes:

[0014] Adding the first frame of video in the video frame sequence as the current frame of video to the key frame sequence;

[0015] Divide the current frame video and the next frame video into a plurality of rectangular blocks;

[0016] Extracting a color histogram of each rectangular block corresponding to the current frame video and the next frame video;

[0017] Comparing the color histograms at corresponding positions of the current frame video and the next frame video for similarity, to obtain a similarity result;

[0018] If the similarity is less than a preset threshold, the next frame video is added to the key frame sequence as the current frame video, and the process proceeds to the step of dividing the current frame video and the next frame video adjacent to it into a plurality of rectangular blocks; if the similarity is not less than the preset threshold, it is detected whether the next frame video is the last frame of the key frame sequence. If the next frame video is not the last frame of the key frame sequence, the process proceeds to the step of dividing the current frame video and the next frame video adjacent to it into a plurality of rectangular blocks, until the next frame video is the last frame of the key frame sequence.

[0019] Preferably, the step of extracting the underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm to obtain a 128-dimensional SIFT feature vector corresponding to the key frame of each frame specifically includes:

[0020] The Gaussian difference scale space is established based on the Gaussian convolution kernel:

[0021] L G (x,y,k,σ)=G(x,y,σ)×I(x,y)

[0022] Where, L G (x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel;

[0023] An image pyramid is constructed by calculating Gaussian difference scale spaces at different scales for the video image, wherein the image pyramid is divided into multiple groups of images according to the scales, each group having a plurality of layers of images, wherein a current group of images is obtained by sampling a previous group of images;

[0024] Based on the image pyramid, all extreme points and their coordinate positions in the same scale space are searched through scale invariance;

[0025] Taking each extreme point as the origin, sampling is performed in a preset scale area around it, and the gradient directions of the pixels in the preset scale area are counted using a histogram to obtain a statistical result;

[0026] According to the statistical results, the peak direction of the histogram is taken as the direction of the corresponding extreme value point;

[0027] Constructing a two-dimensional plane coordinate system, wherein the coordinate axes of the two-dimensional plane coordinate system are the directions of the extreme points, and projecting the extreme points onto the two-dimensional plane coordinate system according to the coordinate positions and scales of the extreme points;

[0028] A 16×16 module is established with the extreme point as the midpoint, and the gradient histogram of the pixels in each module is calculated. Each module is divided into 8 sub-blocks of the same scale, and a 16-directional gradient histogram is calculated for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

[0029] Preferably, the method further comprises:

[0030] Adding a timestamp to each key frame in the key frame sequence;

[0031] The key frames and their corresponding video tags and corresponding timestamps are associated and indexed to construct a video database.

[0032] In a second aspect, the present invention further provides a video image structuring system based on a work site, comprising:

[0033] A de-framing module is used to obtain the surveillance video of the operation site, de-frame the surveillance video, and obtain a video frame sequence;

[0034] A key frame module, used to calculate the similarity between two adjacent video frames in the video frame sequence by using a histogram comparison method, extract video frames whose similarity is less than a preset threshold as key frames, and add the key frames to the key frame sequence;

[0035] A feature extraction module, used to extract the underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm, and obtain a 128-dimensional SIFT feature vector corresponding to the key frame of each frame;

[0036] A normalization module, used for normalizing the 128-dimensional SIFT feature vector to obtain a normalized feature vector;

[0037] The classifier module is used to train the pre-acquired historical work site video data based on the discriminant random field model, and use the pre-set semantic category label as the output, so as to build a classifier model;

[0038] A classification module, used for inputting the normalized feature vector into the classifier model, thereby outputting a semantic category label, and obtaining a mapping relationship between the normalized feature vector and the semantic category label;

[0039] The marking module is used to mark the corresponding key frame with a video label according to the mapping relationship between the normalized feature vector and the semantic category label, so as to obtain a monitoring video containing the video label.

[0040] Preferably, the key frame module specifically includes:

[0041] A current frame extraction module, used for adding the first frame of video in the video frame sequence as the current frame of video to the key frame sequence;

[0042] A segmentation module, used for segmenting the current frame video and the next frame video into a plurality of rectangular blocks;

[0043] A histogram extraction module, used to extract the color histogram of each rectangular block corresponding to the current frame video and the next frame video;

[0044] A similarity comparison module is used to compare the color histograms at corresponding positions of the current frame video and the next frame video to obtain a similarity result;

[0045] A discrimination module is used to add the next frame video as the current frame video to the key frame sequence if the similarity is less than a preset threshold; and is also used to detect whether the next frame video is the last frame of the key frame sequence if the similarity is not less than the preset threshold, and if the next frame video is not the last frame of the key frame sequence, notify the segmentation module to work until the next frame video is the last frame of the key frame sequence.

[0046] Preferably, the feature extraction module specifically includes:

[0047] The scale space construction module is used to build the Gaussian difference scale space based on the Gaussian convolution kernel.

[0048] L G (x,y,k,σ)=G(x,y,σ)×I(x,y)

[0049] Where, L G (x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel;

[0050] A pyramid construction module is used to construct an image pyramid by calculating Gaussian difference scale spaces at different scales for the video image, wherein the image pyramid is divided into multiple groups of images according to the scales, each group having several layers of images, wherein a current group of images is obtained by sampling a previous group of images;

[0051] An extreme point search module, used for searching all extreme points and their coordinate positions in the same scale space through scale invariance based on the image pyramid;

[0052] A gradient statistics module is used to take each extreme point as the origin, perform sampling in a preset scale area around it, and use a histogram to count the gradient directions of the pixels in the preset scale area to obtain statistical results;

[0053] A direction specifying module, used for taking the peak direction of the histogram as the direction of the corresponding extreme value point according to the statistical result;

[0054] A projection module is used to construct a two-dimensional plane coordinate system, the coordinate axis of which is the direction of the extreme point, and project the extreme point onto the two-dimensional plane coordinate system according to the coordinate position and scale of the extreme point;

[0055] The feature vector acquisition module is used to establish a 16×16 module with the extreme point as the midpoint, and calculate the gradient histogram of the pixel points in each module, divide each module into 8 sub-blocks of the same scale, and calculate a 16-directional gradient histogram for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

[0056] Preferably, the system further comprises:

[0057] A timestamp module, used for adding a timestamp to each key frame in the key frame sequence;

[0058] The database module is used to associate the key frames with their corresponding video tags and corresponding timestamps and generate indexes to build a video database.

[0059] In a third aspect, the present invention further provides an electronic device, comprising:

[0060] Memory, used to store computer programs;

[0061] The processor is used to implement the above method steps when executing the program stored in the memory.

[0062] In a fourth aspect, the present invention further provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above-mentioned method steps when executed by a processor.

[0063] It can be seen from the above technical solutions that the present invention has the following advantages:

[0064] The present invention obtains a video frame sequence by de-framing a surveillance video, extracts key frames from the video frame sequence by a histogram comparison method, extracts a corresponding 128-dimensional SIFT feature vector from a video image of each key frame based on a SIFT algorithm, normalizes the 128-dimensional SIFT feature vector, constructs a classifier model, inputs the 128-dimensional SIFT feature vector into the classifier model, outputs corresponding semantic category labels, and adds the semantic category labels to corresponding key frames, thereby obtaining a surveillance video containing a video label, establishing a semantic structure of video data, and improving the efficiency and accuracy of interpreting video information. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flowchart of a video image structuring method based on a work site provided by an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of the structure of a video image structuring system based on a work site provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] For easier understanding, see Figure 1 The present invention provides a method for structuring video images based on a work site, comprising the following steps:

[0069] 100. Obtain surveillance video of the work site, deframe the surveillance video, and obtain a video frame sequence.

[0070] Among them, the video frame sequence obtained by de-framing the monitoring video has a time sequence.

[0071] 200. A histogram comparison method is used to calculate the similarity between two adjacent video frames in a video frame sequence, video frames whose similarity is less than a preset threshold are extracted as key frames, and the key frames are added to the key frame sequence.

[0072] It should be noted that this embodiment uses a histogram comparison method to extract key frames from a surveillance video, thereby extracting representative frames from the video to express the video content.

[0073] 300. Based on the SIFT algorithm, the underlying features of the video image of each key frame in the key frame sequence are extracted to obtain a 128-dimensional SIFT feature vector corresponding to each key frame.

[0074] 400. Normalize the 128-dimensional SIFT feature vector to obtain a normalized feature vector.

[0075] It can be understood that normalizing the 128-dimensional SIFT feature vector can further remove the influence of illumination on it.

[0076] 500. Based on the discriminative random field model, the pre-acquired historical work site video data is trained with the pre-set semantic category labels as output, so as to construct a classifier model.

[0077] Among them, historical work site video data can be acquired in advance, and the pre-set semantic category labels can be manually set according to the historical work site video data. In a specific example, the semantic category labels are set to not wearing a safety helmet, crossing a physical fence, a person falling to the ground, and no abnormal situation according to the video content.

[0078] 600. Input the normalized feature vector into a classifier model, thereby outputting a semantic category label and obtaining a mapping relationship between the normalized feature vector and the semantic category label.

[0079] 700. Label the corresponding key frames with video labels according to the mapping relationship between the normalized feature vector and the semantic category label, thereby obtaining a monitoring video containing the video label.

[0080] The present embodiment provides a video image structuring method based on a work site, which obtains a video frame sequence by deframing the monitoring video, extracts key frames from the video frame sequence by using a histogram comparison method, extracts a corresponding 128-dimensional SIFT feature vector from the video image of each key frame based on the SIFT algorithm, and normalizes the 128-dimensional SIFT feature vector. By constructing a classifier model, the 128-dimensional SIFT feature vector is input into the classifier model, and the corresponding semantic category label is output. The semantic category label is added to the corresponding key frame, thereby obtaining a monitoring video containing a video label, so as to establish a semantic structure of the video data, thereby improving the efficiency and accuracy of interpreting video information.

[0081] In a specific embodiment, step 200 specifically includes:

[0082] 201. Add the first frame of video in the video frame sequence as the current frame of video to the key frame sequence.

[0083] The key frame sequence is empty in the initial state, and the first frame of the video is the first frame of the video frame sequence in chronological order.

[0084] 202. Divide the current frame video and the next frame video into a plurality of rectangular blocks.

[0085] In this embodiment, the image is divided into 3*3 rectangular blocks.

[0086] 203. Extract the color histogram of each rectangular block corresponding to the current frame video and the next frame video.

[0087] 204. Perform similarity comparison on the color histograms at corresponding positions of the current frame video and the next frame video to obtain a similarity result.

[0088] If the similarity is less than a preset threshold, the next frame of video is added to the key frame sequence as the current frame of video, and the process goes to step 202; if the similarity is not less than the preset threshold, the process checks whether the next frame of video is the last frame of the key frame sequence. If the next frame of video is not the last frame of the key frame sequence, the process goes to step 202 until the next frame of video is the last frame of the key frame sequence.

[0089] In a specific embodiment, step 300 specifically includes:

[0090] 301. Based on the Gaussian convolution kernel, the Gaussian difference scale space is established as follows:

[0091] L G (x,y,k,σ)=G(x,y,σ)×I(x,y)

[0092] Where, L G(x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel;

[0093] The Gaussian convolution kernel is represented by the existing technology.

[0094] 302. An image pyramid is constructed by calculating Gaussian difference scale spaces at different scales for video images. The image pyramid is divided into multiple groups of images according to the scales. Each group has several layers of images, wherein a current group of images is obtained by sampling a previous group of images.

[0095] 303. Based on the image pyramid, all extreme points and their coordinate positions in the same scale space are searched through scale invariance.

[0096] Among them, the extreme points are feature points with scale invariance.

[0097] 304. Taking each extreme point as the origin, sampling is performed in a preset scale area around it, and the gradient directions of the pixels in the preset scale area are counted using a histogram to obtain a statistical result.

[0098] 305. According to the statistical results, the peak direction of the histogram is taken as the direction of the corresponding extreme point.

[0099] Among them, in addition to obtaining the extreme points through scale invariance, it is also necessary to use the local features of the image to assign a reference direction to each extreme point so that the descriptor is invariant to image rotation.

[0100] 306. Construct a two-dimensional plane coordinate system, where the coordinate axes of the two-dimensional plane coordinate system are the directions of the extreme points, and project the extreme points onto the two-dimensional plane coordinate system according to the coordinate positions and scales of the extreme points.

[0101] 307. A 16×16 module is established with the extreme point as the midpoint, and the gradient histogram of the pixels in each module is calculated. Each module is divided into 8 sub-blocks of the same scale, and a 16-directional gradient histogram is calculated for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

[0102] In a specific embodiment, the method further comprises:

[0103] 800. Add a timestamp to each key frame in the key frame sequence;

[0104] It should be noted that the timestamp is added based on the actual acquisition time of the key frame.

[0105] 900. Associating key frames, corresponding video tags and corresponding timestamps, generating indexes, and constructing a video database.

[0106] By creating an index, each index corresponds to a timestamp and its corresponding key frame, video tag and video content, so that the corresponding video content can be found through the video tag.

[0107] The above is a detailed description of an embodiment of a video image structuring method based on a work site provided by the present invention. The following is a detailed description of an embodiment of a video image structuring system based on a work site provided by the present invention.

[0108] For easier understanding, see Figure 2 The present invention provides a video image structuring system based on a work site, comprising:

[0109] The de-framing module 10 is used to obtain the monitoring video of the operation site, de-frame the monitoring video, and obtain a video frame sequence;

[0110] A key frame module 20 is used to calculate the similarity between two adjacent video frames in a video frame sequence by using a histogram comparison method, extract video frames whose similarity is less than a preset threshold as key frames, and add the key frames to the key frame sequence;

[0111] The feature extraction module 30 is used to extract the underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm to obtain a 128-dimensional SIFT feature vector corresponding to each key frame;

[0112] A normalization module 40 is used to normalize the 128-dimensional SIFT feature vector to obtain a normalized feature vector;

[0113] The classifier module 50 is used to train the pre-acquired historical work site video data based on the discriminant random field model, using the pre-set semantic category label as the output, so as to construct a classifier model;

[0114] A classification module 60, used for inputting the normalized feature vector into a classifier model, thereby outputting a semantic category label, and obtaining a mapping relationship between the normalized feature vector and the semantic category label;

[0115] The marking module 70 is used to mark the corresponding key frames with video labels according to the mapping relationship between the normalized feature vector and the semantic category label, so as to obtain the monitoring video containing the video label.

[0116] In a specific embodiment, the key frame module specifically includes:

[0117] A current frame extraction module, used for adding the first frame of video in the video frame sequence as the current frame of video to the key frame sequence;

[0118] A segmentation module, used for segmenting the current frame video and the next frame video into a number of rectangular blocks;

[0119] A histogram extraction module is used to extract the color histogram of each rectangular block corresponding to the current frame video and the next frame video;

[0120] A similarity comparison module is used to compare the color histograms at the corresponding positions of the current frame video and the next frame video to obtain a similarity result;

[0121] The discrimination module is used to add the next frame video as the current frame video to the key frame sequence if the similarity is less than a preset threshold; and is also used to detect whether the next frame video is the last frame of the key frame sequence if the similarity is not less than the preset threshold. If the next frame video is not the last frame of the key frame sequence, the segmentation module is notified to work until the next frame video is the last frame of the key frame sequence.

[0122] In a specific embodiment, the feature extraction module specifically includes:

[0123] The scale space construction module is used to establish the Gaussian difference scale space based on the Gaussian convolution kernel.

[0124] L G (x,y,k,σ)=G(x,y,σ)×I(x,y)

[0125] Where, L G (x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel;

[0126] A pyramid construction module is used to construct an image pyramid by calculating Gaussian difference scale spaces at different scales for video images. The image pyramid is divided into multiple groups of images according to the scales, and each group has several layers of images, wherein the current group of images is obtained by sampling the previous group of images;

[0127] The extreme point search module is used to search for all extreme points and their coordinate positions in the same scale space based on the image pyramid through scale invariance;

[0128] The gradient statistics module is used to take each extreme point as the origin, perform sampling in the preset scale area around it, and use the histogram to count the gradient direction of the pixels in the preset scale area to obtain the statistical result;

[0129] A direction specifying module is used to take the peak direction of the histogram as the direction of the corresponding extreme point according to the statistical results;

[0130] A projection module is used to construct a two-dimensional plane coordinate system, the coordinate axis of which is the direction of the extreme point, and the extreme point is projected onto the two-dimensional plane coordinate system according to the coordinate position and scale of the extreme point;

[0131] The feature vector acquisition module is used to establish a 16×16 module with the extreme point as the midpoint, and calculate the gradient histogram of the pixels in each module, divide each module into 8 sub-blocks of the same scale, and calculate a 16-directional gradient histogram for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

[0132] The present embodiment provides a video image structuring system based on a work site, which obtains a video frame sequence by deframing the monitoring video, extracts key frames from the video frame sequence by using a histogram comparison method, extracts the corresponding 128-dimensional SIFT feature vector of the video image of each key frame based on the SIFT algorithm, and normalizes the 128-dimensional SIFT feature vector. By constructing a classifier model, the 128-dimensional SIFT feature vector is input into the classifier model, and the corresponding semantic category label is output. The semantic category label is added to the corresponding key frame, thereby obtaining a monitoring video containing a video label, so as to establish a semantic structure of the video data, thereby improving the efficiency and accuracy of interpreting video information.

[0133] In a specific embodiment, the system further includes:

[0134] A timestamp module is used to add a timestamp to each key frame in a key frame sequence;

[0135] The database module is used to associate key frames with their corresponding video tags and corresponding timestamps and generate indexes to build a video database.

[0136] The present invention also provides an electronic device, comprising:

[0137] Memory, used to store computer programs;

[0138] The processor is used to implement the method steps in the above embodiment when executing the program stored in the memory.

[0139] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method steps in the above embodiment are implemented.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present invention through a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0145] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video image structuring method based on a work site, characterized in that: The following steps are involved: Obtaining a surveillance video of the work site, de-framing the surveillance video, and obtaining a video frame sequence; The similarity between two adjacent video frames in the video frame sequence is calculated by using a histogram comparison method, a video frame whose similarity is less than a preset threshold is extracted as a key frame, and the key frame is added to the key frame sequence; Extracting underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm to obtain a 128-dimensional SIFT feature vector corresponding to the key frame of each frame; Normalizing the 128-dimensional SIFT feature vector to obtain a normalized feature vector; The previously acquired historical work site video data is trained based on the discriminative random field model, and the pre-set semantic category labels are used as output to build a classifier model; wherein the semantic category labels are set as not wearing a safety helmet, crossing a physical fence, a person falling to the ground, and no abnormal situation; Inputting the normalized feature vector into the classifier model, thereby outputting a semantic category label, and obtaining a mapping relationship between the normalized feature vector and the semantic category label; The corresponding key frames are marked with video labels according to the mapping relationship between the normalized feature vector and the semantic category label, so as to obtain a monitoring video containing the video label.

2. The method for structuring video images based on a work site according to claim 1, characterized in that: The step of using the histogram comparison method to calculate the similarity between two adjacent video frames in the video frame sequence, extracting video frames whose similarity is less than a preset threshold as key frames, and adding the key frames to the key frame sequence specifically includes: Adding the first frame of video in the video frame sequence as the current frame of video to the key frame sequence; Divide the current frame video and the next frame video into a plurality of rectangular blocks; Extracting a color histogram of each rectangular block corresponding to the current frame video and the next frame video; Comparing the color histograms at corresponding positions of the current frame video and the next frame video for similarity, to obtain a similarity result; If the similarity is less than a preset threshold, the next frame video is added to the key frame sequence as the current frame video, and the process proceeds to the step of dividing the current frame video and the next frame video adjacent to it into a plurality of rectangular blocks; if the similarity is not less than the preset threshold, it is detected whether the next frame video is the last frame of the key frame sequence. If the next frame video is not the last frame of the key frame sequence, the process proceeds to the step of dividing the current frame video and the next frame video adjacent to it into a plurality of rectangular blocks, until the next frame video is the last frame of the key frame sequence.

3. The video image structuring method based on the work site according to claim 1 is characterized in that: The step of extracting the underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm to obtain a 128-dimensional SIFT feature vector corresponding to each key frame specifically includes: The Gaussian difference scale space is established based on the Gaussian convolution kernel: L G (x,y,k,σ)=G(x,y,σ)×I(x,y) Where, L G (x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel; An image pyramid is constructed by calculating Gaussian difference scale spaces at different scales for the video image, wherein the image pyramid is divided into multiple groups of images according to the scales, each group having a plurality of layers of images, wherein a current group of images is obtained by sampling a previous group of images; Based on the image pyramid, all extreme points and their coordinate positions in the same scale space are searched through scale invariance; Taking each extreme point as the origin, sampling is performed in a preset scale area around it, and the gradient directions of the pixels in the preset scale area are counted using a histogram to obtain a statistical result; According to the statistical results, the peak direction of the histogram is taken as the direction of the corresponding extreme value point; Constructing a two-dimensional plane coordinate system, wherein the coordinate axes of the two-dimensional plane coordinate system are the directions of the extreme points, and projecting the extreme points onto the two-dimensional plane coordinate system according to the coordinate positions and scales of the extreme points; A 16×16 module is established with the extreme point as the midpoint, and the gradient histogram of the pixels in each module is calculated. Each module is divided into 8 sub-blocks of the same scale, and a 16-directional gradient histogram is calculated for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

4. The method for structuring video images based on a work site according to claim 1, characterized in that: Also includes: Adding a timestamp to each key frame in the key frame sequence; The key frames and their corresponding video tags and corresponding timestamps are associated and indexed to construct a video database.

5. A video image structuring system based on a work site, characterized in that: include: A de-framing module is used to obtain the surveillance video of the operation site, de-frame the surveillance video, and obtain a video frame sequence; A key frame module, used to calculate the similarity between two adjacent video frames in the video frame sequence by using a histogram comparison method, extract video frames whose similarity is less than a preset threshold as key frames, and add the key frames to the key frame sequence; A feature extraction module, used to extract the underlying features of the video image of each key frame in the key frame sequence based on the SIFT algorithm, and obtain a 128-dimensional SIFT feature vector corresponding to the key frame of each frame; A normalization module, used for normalizing the 128-dimensional SIFT feature vector to obtain a normalized feature vector; The classifier module is used to train the pre-acquired historical work site video data based on the discriminative random field model, and use the pre-set semantic category labels as output quantities to build a classifier model; wherein the semantic category labels are set as not wearing a safety helmet, crossing a physical fence, a person falling to the ground, and no abnormal situation; A classification module, used for inputting the normalized feature vector into the classifier model, thereby outputting a semantic category label, and obtaining a mapping relationship between the normalized feature vector and the semantic category label; The marking module is used to mark the corresponding key frame with a video label according to the mapping relationship between the normalized feature vector and the semantic category label, so as to obtain a monitoring video containing the video label.

6. The video image structuring system based on the work site according to claim 5 is characterized in that: The key frame module specifically includes: A current frame extraction module, used for adding the first frame of video in the video frame sequence as the current frame of video to the key frame sequence; A segmentation module, used for segmenting the current frame video and the next frame video into a plurality of rectangular blocks; A histogram extraction module, used to extract the color histogram of each rectangular block corresponding to the current frame video and the next frame video; A similarity comparison module is used to compare the color histograms at corresponding positions of the current frame video and the next frame video to obtain a similarity result; A discrimination module is used to add the next frame video as the current frame video to the key frame sequence if the similarity is less than a preset threshold; and is also used to detect whether the next frame video is the last frame of the key frame sequence if the similarity is not less than the preset threshold, and if the next frame video is not the last frame of the key frame sequence, notify the segmentation module to work until the next frame video is the last frame of the key frame sequence.

7. The video image structuring system based on the work site according to claim 5, characterized in that: The feature extraction module specifically includes: The scale space construction module is used to build the Gaussian difference scale space based on the Gaussian convolution kernel. L G (x,y,k,σ)=G(x,y,σ)×I(x,y) Where, L G (x, y, k, σ) represents the Gaussian difference scale space, (x, y) represents the image coordinates of the pixel point, k represents the constant factor of the difference between two adjacent scale spaces, σ represents the scale space factor, I(x, y) represents the video image, and G(x, y, σ) represents the Gaussian convolution kernel; A pyramid construction module is used to construct an image pyramid by calculating Gaussian difference scale spaces at different scales for the video image, wherein the image pyramid is divided into multiple groups of images according to the scales, each group having several layers of images, wherein a current group of images is obtained by sampling a previous group of images; An extreme point search module, used for searching all extreme points and their coordinate positions in the same scale space through scale invariance based on the image pyramid; A gradient statistics module is used to take each extreme point as the origin, perform sampling in a preset scale area around it, and use a histogram to count the gradient directions of the pixels in the preset scale area to obtain statistical results; A direction specifying module, used for taking the peak direction of the histogram as the direction of the corresponding extreme value point according to the statistical result; A projection module is used to construct a two-dimensional plane coordinate system, the coordinate axis of which is the direction of the extreme point, and project the extreme point onto the two-dimensional plane coordinate system according to the coordinate position and scale of the extreme point; The feature vector acquisition module is used to establish a 16×16 module with the extreme point as the midpoint, and calculate the gradient histogram of the pixel points in each module, divide each module into 8 sub-blocks of the same scale, and calculate a 16-directional gradient histogram for each sub-block, thereby forming 8 16-directional vectors to obtain a 128-dimensional SIFT feature vector.

8. The video image structuring system based on the work site according to claim 5, characterized in that: Also includes: A timestamp module, used for adding a timestamp to each key frame in the key frame sequence; The database module is used to associate the key frames with their corresponding video tags and corresponding timestamps and generate indexes to build a video database.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images