Method for establishing video monitoring streaming audio and video storage semantic index based on multiple modes

Abstract

The invention provides a method for establishing a video monitoring streaming audio and video storage semantic index based on multiple modes, and belongs to the field of audio and video processing. The problem of low audio and video to semantic index conversion efficiency is solved; the method specifically comprises the following steps: S1, obtaining a to-be-extracted image, a to-be-extracted audio and text information; s2, extracting an image feature set of the to-be-extracted image, and extracting a voice feature set of the to-be-extracted audio; s3, fusing the image feature set and the voice feature set to obtain fused features; concluding and classifying the to-be-extracted image and the to-be-extracted audio according to the fusion features, and establishing semantic annotations in combination with the text information; s4, summarizing semantic annotations and feeding back the semantic annotations; according to the method, the semantic index is established by acquiring, processing and summarizing the audios and the videos, so that the efficiency of converting the audios and the videos into the semantic index is improved.

Description

Technical Field

[0001] The invention discloses a method for establishing a video surveillance streaming audio and video storage semantic index based on multi-modality, and relates to the field of audio and video processing. Background Art

[0002] The existing semantic indexing methods for audio and video storage have the following shortcomings: High computing requirements: Existing semantic indexing methods require a lot of computing resources when extracting image and speech features. At the same time, the image or speech features extracted by existing methods are not clearly targeted. Image and speech features that are too broad or too narrow will make the final feature fusion analysis results wrong, which is not conducive to the establishment of semantic indexing.

[0003] Sensitive to data quality: Existing semantic indexing methods do not preprocess the video and audio or have few preprocessing steps for the video and audio, resulting in poor quality of input video and audio data (such as blur, noise, occlusion, etc.), which affects the effect of multimodal feature extraction and semantic analysis.

[0004] Semantic ambiguity: Existing semantic indexing methods have misunderstandings when processing videos and audios with complex content. When processing some vague or ambiguous video and audio content, they are unable to make accurate semantic analysis and indexing. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method for establishing a semantic index for video surveillance streaming audio and video storage based on multimodality, aiming to solve the problem of low efficiency in converting audio and video to semantic index.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solution: A method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality includes: Step S1: Obtain a surveillance video of a target area; extract a video portion and a voice information portion from the surveillance video to obtain an original video and original voice information; Extract and enhance the frame images in the original video to obtain the image to be extracted; Extract and enhance the audio stream in the original voice information to obtain the audio and text information to be extracted; Step S2: extracting color, shape, texture and edge features from the image to be extracted to obtain an image feature set; Calculate the MFCC coefficients of the audio to be extracted, perform homomorphic and cepstrum processing on each voice information in the audio to be extracted, calculate the LPCC coefficients of each voice information, and obtain a voice feature set; Step S3: performing multimodal feature fusion according to the image feature set and the speech feature set to obtain fusion features; using a machine learning algorithm, summarizing and classifying the image to be extracted and the audio to be extracted according to the fusion features, and establishing semantic annotations in combination with text information; Step S4: Summarize semantic annotations and provide feedback.

[0007] Furthermore, the specific steps of step S1 are as follows: Step S11: Obtain the surveillance video of the target area and obtain the absolute path of the surveillance video; Step S12: extracting the image part in the surveillance video to obtain the original video; Extract the voice information part in the surveillance video to obtain the original voice information; Step S13: extracting and enhancing the frame image in the original video to obtain an image to be extracted; Get the resolution of the original video, recorded as w×l; w represents the pixel length of the original video, l represents the pixel width of the original video; get the number of frames of the original video, recorded as ts; De-noising, tilt correction and normalization operations are performed on the first frame image in the original video to obtain a standard image 1; Step S14: Repeat the same step of processing the first frame image in the original video to obtain the standard image 1, and process the second to ts-th frame images in the original video to obtain standard images 2 to standard images ts; Use standard image 1 to standard image ts as images to be extracted; Using OSR technology, scan and extract visible texts in standard images 1 to standard images ts to obtain text α; Step S15: Perform pre-emphasis, windowing and Fourier transform enhancement processing on the original voice information to obtain the audio and text information to be extracted, and then go to step S2.

[0008] Furthermore, the specific steps of step S13 are as follows: Step S131: record the first frame image in the original video as image A; Let the square root of (w×l) be pa; perform continuous square root operations on (w×l) until the value of pa is between (0,10); pa is a positive integer, rounded down; Step S132: With a convolution kernel size of pa×pa, use the cv2.GaussianBlur function in the OpenCV library to perform Gaussian blur denoising on image A to obtain image A1; Use cv2.cvtHOlor() method to convert image A1 into a grayscale image, and use cv2.threshold() method to convert the grayscale image into a binary image to obtain image A2; Step S133: Use cv2.findHOntours() method to find the largest contour in image A2, use the minimum enclosing rectangle method to calculate the rotation angle of the largest contour, use cv2.getRotationMatrix2D and cv2.warpAffine function to rotate image A2 to obtain image A3; Step S134: Read the grayscale value of each pixel in image A3, denoted as HO （1,1） , HO （1,2） ~HO （w,l） ; Among them, HO （1,1） Indicates the gray value corresponding to the pixel at (1,1) in image A3; HO （1,2） Indicates the gray value corresponding to the pixel at (1,2) in image A3; and so on, HO （w,l） Represents the gray value corresponding to the pixel at (w, l) in image A3; Step S135: extract the maximum value of the pixel grayscale in image A3 and record it as R MAX , the minimum value is denoted as R MIN ; Calculate the average grayscale value of all pixels in image A3, denoted as R AVE ; Normalize the pixel at (1,1) in image A3 and calculate HO （1,1） The standardized value of （1,1） ; Determine HO （1,1） Is it in R MAX ~R AVE between; If yes, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(HO （1,1） -R AVE ) / (R MAX -R AVE )]×256} / 2-1;CB （1,1） Round up; If not, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(R AVE -HO （1,1） ) / (R AVE -R MIN )]×256} / 2-1;CB （1,1） Round down; Step S136: Repeat calculation of HO （1,1） The same procedure as the normalized values ​​is used to calculate HO （1,2） ~HO（w,l） The standardized value of is used to complete the standardization of image A3 and obtain the standard image 1.

[0009] Furthermore, the specific steps of step S15 are as follows: Step S151: obtaining the duration of the original voice information, denoted as vd; Calculate the rated number of samples of the original speech information, denoted as sq; sq calculation formula: sq = [log 60 (vd)] × 1024; Calculate the frame duration of a single voice message, recorded as fd; Calculation formula of fd: fd = vd we / sq; Calculate the frame shift duration of a single voice message, denoted as fsd; The calculation formula of fsd: fsd = fd / 2; Step S152: Record the original voice information as information V; Use a high-pass filter to remove low-frequency noise in information V to obtain information V1; The information V1 is framed by the frame length fd and the frame shift length fsd to obtain the information V2; The amount of information V2 is recorded as F, F = vd / fd; F is a positive number, rounded up; Step S153: Using a Hamming window to perform windowing processing on the information V2 to obtain information V3; the number of information V3 is the same as the number of information V2; The function expression w(n) of the Hamming window is as follows: w(n)=0.54-0.46cos[[2π×(n-1)] / F)]; wherein n is a positive integer, n represents the index of the nth information V2, and the value range of n is: 0-F; w(n) represents the function expression for converting the nth information V2 into the nth information V3; Step S154: using Fourier transform, converting the information V3 of each timestamp into the frequency domain information of the corresponding timestamp to obtain information V4; the number of information V4 is the same as the number of information V3; The functional expression of Fourier transform is as follows: ; Wherein, k represents the index of information V4, n represents the index of information V3, and the value range of k and n is: 0~F; X(k) represents the frequency domain information of the kth timestamp, and x(n) represents the time domain information of the nth timestamp; e represents a natural constant, j represents an imaginary unit, and π represents the ratio of circumference to circumference; Step S155: Summarize the information V4 corresponding to the 1st to nth timestamps as the audio to be extracted; Use the SpeechRecognition library and Google Web Speech API to convert the audio to be extracted into text to obtain text β; Combine text α and text β as text information.

[0010] Furthermore, the specific steps of step S2 are as follows: Step S21: using the standard image 1 as image A; Extract color, shape, texture and edge features from image A; Step S22: Repeat the same steps of extracting the color, shape, texture and edge features of the standard image 1 to extract the color, shape, texture and edge features corresponding to the standard images 2 to ts; Summarize the color, shape, texture and edge features corresponding to the standard images 1 to ts as an image feature set; Step S23: Calculate the MFCC coefficients of the audio to be extracted, and perform homomorphic and cepstrum processing on each voice information in the audio to be extracted to obtain the LPCC coefficients of each voice information; Step S231: Obtain the total amount F of voice information in the audio to be extracted; The voice information in the audio to be extracted is recorded as information V5; Get the frequency of each message V5, denoted as f 1 、f 2 ~f F ; Among them, f 1 Indicates the frequency of information V5 corresponding to the first timestamp; f 2 Indicates the frequency of information V5 corresponding to the second timestamp; and so on, f F Indicates the frequency of information V5 corresponding to the Fth timestamp; Step S232: Calculate the MFCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S2321: Convert the V5 frequency of the information corresponding to each timestamp into the Mel frequency. The conversion formula of the Mel frequency is as follows: F mel (f i ) = 2595 × lg (1 + f i / 700); where i represents the index of information V5, f i represents the frequency of the information V5 corresponding to the i-th timestamp; F mel (f i ) indicates f in Mel iThe perceived frequency (i.e. Mel frequency); Substitute the frequency of the information V5 from the 1st to the Fth timestamp into the Mel frequency conversion formula, calculate the perceived frequency of the 1st to the Fth timestamp, and get F mel (f 1 )~F mel (f F ); Step S2322: performing discrete cosine transform on the perceived frequency of the information V6 to obtain the MFCC coefficients of the audio A to be extracted; The formula for discrete cosine transform is as follows: ; Among them, MFCC (i) represents the MFCC coefficient corresponding to the i-th information V5; F mel (f 1 )~F mel (f F ) is substituted into the discrete cosine transform formula to calculate the MFCC coefficients corresponding to the 1st to Fth timestamps, and MFCC (1) to MFCC (F) are obtained; Step S233: Calculate the LPCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S234: Summarize the MFCC coefficients and LPCC coefficients of the audio to be extracted as the speech feature set of the audio to be extracted, and proceed to step S3.

[0011] Furthermore, the specific steps of step S21 are as follows: Step S211: Obtain pixels of image A, denoted as w×l, where w represents pixel length and l represents pixel width; Use the Pillow library function to get the RGB value of each pixel in image A and get the RGB matrix: ; Among them, X 11 Indicates the RGB value corresponding to the pixel in the first row and first column of image A; X 12 represents the RGB value corresponding to the pixel in the first row and second column of image A; and so on, X wl Indicates the RGB value corresponding to the pixel in the wth row and the lth column in image A; Calculate X 11 ~X wl The average value of is taken as the pixel average value of image A, recorded as aX; calculate X 11 ~X wl The standard deviation of is taken as the pixel standard deviation of image A, denoted as μX; Step S212: Calculate the skewness of each color channel in image A: Calculate the average RGB value of all pixels in the first color channel, denoted as aX(1); the calculation formula of aX(1) is as follows: ; Where j is a positive integer, and the value range of j is: 1~1; X 1j Indicates the RGB value corresponding to the pixel in the 1st row and the jth column in image A; Calculate the skewness of the first color channel, denoted as SS(1); the calculation formula of SS(1) is as follows: ; Calculate the average RGB value of all pixels in the second color channel, denoted as aX(2); the calculation formula of aX(2) is as follows: ; Among them, X 2j Indicates the RGB value corresponding to the pixel in the 2nd row and the jth column in image A; Calculate the skewness of the second color channel, denoted as SS(2); the calculation formula of SS(2) is as follows: ; Similarly, the average RGB value of all pixels in the w-th color channel is calculated, denoted as aX(w); the calculation formula of aX(w) is as follows: ; Among them, X wj Indicates the RGB value corresponding to the pixel in the wth row and the jth column in image A; Calculate the skewness of the w-th color channel, denoted as SS(w); the calculation formula of SS(w) is as follows: ; Calculate the average value of SS(1)~SS(w) as the total skewness of image A, denoted as aSS, and take aX, μX and aSS as the color features of image A; Step S213: In the RGB matrix, read the maximum value of the pixel RGB value and record it as X MAX , the minimum value is recorded as X MIN ; Calculate the average value of pixel RGB value and record it as X AVE ; Perform secondary normalization on image A; Define the calculation process of secondary normalization of each pixel in image A: Let the RGB value of the pixel in the i-th row and j-th column of image A be X ij ;X ij The standardized value of ij ; Judge X ij Is it in X MAX ~X AVE Between, determine CX ij Calculation formula of If X ij In X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1+(X ij -X AVE ) / (X MAX -X AVE )]×128-1; If X ij Not in X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1-(X AVE -X ij ) / (X AVE -X MIN )]×128-1; Step S214: Calculate the standard value of each pixel in the picture A to obtain a binary image of the picture A; Use OpenCV library to extract shape features of binary images; Step S2141: using the Canny edge detection algorithm to find the edge of the binary image; Locate the image based on the edge of the binary image and use the cv2.findContours function to find the contours of each object in the binary image; Step S2142: For the contour of each object in the binary image, use the arcLength() function to calculate the length of the contour; Use the contourArea() function to calculate the area of ​​the contour; Use the boundingRect() function to calculate the bounding box of the contour; Use the approxPolyDP() function to calculate the approximate polygon of the contour; Step S2143: Summarize the contour length, contour area, contour bounding box and contour approximate polygon of each object in the binary image as the shape feature of the image A; Step S215: define a 256×256 empty matrix as an initial gray-level co-occurrence matrix; 1 is the distance between pixels, 0 is o The direction is 256 gray levels. Use the greycomatrix function to scan the binary image and fill in the initial gray co-occurrence matrix to obtain the gray co-occurrence matrix of the image set to be extracted, which is recorded as the co-occurrence matrix A. The number of pixel pairs in the co-occurrence matrix A is counted, denoted as pip; the angular second-order moment of each pixel pair in the co-occurrence matrix A and the diagonal mean square difference corresponding to the 1st to pipth pixels are calculated using Python and NumPy libraries as the texture features of image A.

[0012] Furthermore, the specific steps of step S233 are as follows: Step S2331: Perform Z transformation on the 1st to Fth information V5 to obtain Z(1)~Z(F); The function expression of Z transform is as follows: ; where i represents the number of information V5, f i represents the frequency of information V5 corresponding to the i-th timestamp; Z(i) represents the output result of Z-transformation on the i-th information V5; z i The complex variable representing the i-th information V5, z i The calculation formula is as follows: z i = (vc × f i )×e(j×ω) -1 ; where vc represents the speed of sound, e represents a natural constant, and j represents an imaginary unit (i.e. j 2 =-1); ω represents the phase of the i-th information E, and the calculation formula of ω is as follows: ω=arctan[(vc×f i ) / fd]; where arctan represents the inverse tangent function; f 1 ~f F Substitute into the function expression of Z transform, perform Z transform on the 1st to Fth information V5, and obtain Z(1)~Z(F); Step S2332: Perform inverse Z transform on Z(1) to Z(F); The function expression of logarithmic transformation is as follows: ; Wherein, lnZ(i) represents the output result of logarithmic transformation of Z(i); Substitute Z(1)~Z(F) into the function expression of logarithmic transformation, perform logarithmic transformation on the 1st to Fth information V5, and obtain lnZ(1)~lnZ(F); Use the iztrans function to perform inverse Z transform on lnZ(1)~lnZ(F), and transform the information V5 from frequency domain information to time domain information to obtain ts 1 ,ts 2 ~ts F ; Among them, ts 1 Indicates the time domain information of the first timestamp corresponding information V5; ts2 Indicates the time domain information of the second timestamp corresponding to information V5; and so on, ts F Indicates the time domain information of the information V5 corresponding to the Fth timestamp; Step S2333: ts 1 ~ts F Perform cepstrum processing to obtain the LPC coefficients of the 1st to Fth information E, denoted as ce 1 ~ce F ; The function expression of cepstrum processing is as follows: ; Wherein, i and k both represent the number of information E, and the value range of i and k is: 1~F; f i Indicates the frequency of the information V5 corresponding to the i-th timestamp, ts i Represents the time domain information of the information V5 corresponding to the i-th timestamp; Using the discrete cosine transform algorithm, ce 1 ~ce F Convert the time domain information into frequency domain information to obtain the LPCC coefficients of the audio to be extracted.

[0013] Furthermore, the specific steps of step S3 are as follows: Step S31: Summarize the image feature set of each image to be extracted and summarize the voice feature set of the audio to be extracted; Use the image_features object, audio_features object, and text_features object to receive the image feature set, the speech feature set, and the text file in turn to obtain a one-dimensional feature vector; Use NumPy's concatenate function to concatenate the image_features object and the audio_features object to get the concatenated_feature object; Use NumPy's concatenate function to concatenate the concatenated_feature object and the audio_features object to obtain the concatenated_features object; Use the concatenated_features object as the fusion feature; Step S32: Use the ResNet-50 model and the ImageNet dataset to summarize and classify the images and audios to be extracted, and establish semantic annotations in combination with text information.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Multimodal fusion: The present invention can more comprehensively understand and describe the video content by fusing information of multiple modalities such as images, audio, and text in the video. The fusion strategy of the present invention can better capture the key information in the video than single modality analysis, and improve the accuracy and comprehensiveness of the index.

[0015] Efficient retrieval and rapid response: By establishing a semantic-based index, the present invention allows users to perform efficient retrieval based on video content (such as events, scenes, objects, etc.) rather than relying on file names or simple tags; this indexing method makes the retrieval process more intuitive and rapid, meeting the needs of real-time monitoring and rapid response.

[0016] Streaming processing: The present invention can process and index streaming audio and video data in real time, ensuring that newly generated video content can be immediately included in the retrieval scope; the multimodal semantic indexing method combines computer vision, image processing and audio processing technology to achieve intelligent analysis of video content and improve the accuracy and efficiency of indexing; at the same time, automated processing reduces manual intervention, reduces labor costs and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a schematic diagram of the method of the present invention; Figure 2 It is a schematic diagram of the logic processing of the present invention; Figure 3 This is a schematic diagram of data processing of the present invention. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] See also Figure 1 , a method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality includes: Step S1: Obtain a surveillance video of a target area; extract a video portion and a voice information portion from the surveillance video to obtain an original video and original voice information; Extract and enhance the frame images in the original video to obtain the image to be extracted; Extract and enhance the audio stream in the original voice information to obtain the audio and text information to be extracted; It should be noted that the "target area" in the present invention refers to the area where the semantic index is established by using the present invention (a method for establishing a semantic index for video surveillance streaming audio and video storage based on multimodality); The specific steps of step S1 are as follows: Step S11: Obtain the surveillance video of the target area, and obtain the absolute path where the surveillance video is saved by the target area (person in charge); Step S12: extracting the image part in the surveillance video to obtain the original video; Extract the voice information part in the surveillance video to obtain the original voice information; Step S121: extracting an image portion from a surveillance video; With the absolute path as a parameter, use the cv2.VideoCapture() method in the OpenCV library to read the video file in the surveillance video; (OpenCV is an open source computer vision and machine learning software library) Step S122: Use the read() method to read the image in the video file frame by frame, and use the cv2.imwrite method to process the output of the read() method to obtain the original video; Step S123: extracting the voice information part in the surveillance video; With the absolute path as a parameter, use the VideoFileClip() method in the moviepy library to read the video file in the surveillance video and return a VideoFileClip object, recorded as the object video; (moviepy is a Python library for video editing) Step S124: extract the audio attribute of the object video, obtain the voice information in the video file, and receive it with the variable adio; Use the write_audiofile() method to process the variable adio to get the original voice information; Step S13: extracting and enhancing the frame image in the original video to obtain an image to be extracted; Get the resolution of the original video, recorded as w×l, where w represents the pixel length of the original video and l represents the pixel width of the original video; Get the number of frames of the original video, recorded as ts; (the number of frames of the original video = the frame rate of the original video × the duration of the original video) De-noising, tilt correction and normalization operations are performed on the first frame image in the original video to obtain a standard image 1; Step S131: record the first frame image in the original video as image A; Let the square root of (w×l) be pa; perform continuous square root operations on (w×l) until the value of pa is between (0,10); pa is a positive integer, rounded down; Step S132: With a convolution kernel size of pa×pa, use the cv2.GaussianBlur function in the OpenCV library to perform Gaussian blur denoising on image A to obtain image A1; Use cv2.cvtHOlor() method to convert image A1 into a grayscale image, and use cv2.threshold() method to convert the grayscale image into a binary image to obtain image A2; Step S133: Use cv2.findHOntours() method to find the largest contour in image A2, use the minimum enclosing rectangle method to calculate the rotation angle of the largest contour, use cv2.getRotationMatrix2D and cv2.warpAffine function to rotate image A2 to obtain image A3; Step S134: Read the grayscale value of each pixel in image A3, denoted as HO （1,1） , HO （1,2） ~HO （w,l） ; Among them, HO （1,1） Indicates the gray value corresponding to the pixel at (1,1) in image A3; HO （1,2） Indicates the gray value corresponding to the pixel at (1,2) in image A3; and so on, HO （w,l） Represents the gray value corresponding to the pixel at (w, l) in image A3; Step S135: extract the maximum value of the pixel grayscale in image A3 and record it as R MAX , the minimum value is denoted as R MIN ; Calculate the average grayscale value of all pixels in image A3, denoted as R AVE ; Normalize the pixel at (1,1) in image A3 and calculate HO （1,1） The standardized value of （1,1） ; Determine HO （1,1） Is it in R MAX ~R AVE between; If yes, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(HO （1,1） -R AVE ) / (R MAX -R AVE )]×256} / 2-1;CB （1,1） Round up; If not, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(R AVE-HO （1,1） ) / (R AVE -R MIN )]×256} / 2-1;CB （1,1） Round down; Step S136: Repeat calculation of HO （1,1） The same procedure as the normalized value (i.e., step S135) is used to calculate HO （1,2） ~HO （w,l） The standardized value of A3 is used to complete the standardization of image A3 and obtain the standard image 1; Step S14: Repeat the same step of processing the first frame image in the original video to obtain the standard image 1, and process the second to ts-th frame images in the original video to obtain standard images 2 to standard images ts; Use standard image 1 to standard image ts as images to be extracted; Using OSR technology, scan and extract visible texts in standard images 1 to standard images ts to obtain text α; It should be noted that the “visible text” in step S14 of the present invention refers to the subtitle part in the standard image 1 to the standard image ts; if there is no subtitle in the standard image 1 to the standard image ts, the process of obtaining the text α is skipped.

[0020] Step S15: performing pre-emphasis, windowing and Fourier transform enhancement processing on the original voice information to obtain the audio and text information to be extracted; Step S151: obtaining the duration of the original voice information, denoted as vd; Calculate the rated number of samples of the original speech information, denoted as sq; sq calculation formula: sq = [log 60 (vd)] × 1024; Calculate the frame duration of a single voice message, recorded as fd; Calculation formula of fd: fd = vd we / sq; Calculate the frame shift duration of a single voice message, denoted as fsd; The calculation formula of fsd: fsd = fd / 2; Step S152: Record the original voice information as information V; Use a high-pass filter to remove low-frequency noise in information V to obtain information V1; The information V1 is framed by the frame length fd and the frame shift length fsd to obtain the information V2; The number of information V2 (i.e., the number of samples) is recorded as F, F = vd / fd; F is a positive number, rounded up; Step S153: Using a Hamming window to perform windowing processing on the information V2 to obtain information V3; the number of information V3 is the same as the number of information V2; The function expression w(n) of the Hamming window is as follows: w(n)=0.54-0.46cos[[2π×(n-1)] / F)]; wherein n is a positive integer, n represents the index of the nth information V2, and the value range of n is: 0-F; w(n) represents the function expression for converting the nth information V2 into the nth information V3; Step S154: using Fourier transform, convert the information V3 (time domain information) of each timestamp into the frequency domain information of the corresponding timestamp to obtain information V4; the number of information V4 is the same as the number of information V3; The functional expression of Fourier transform is as follows: ; Wherein, k represents the index of information V4 (i.e., frequency domain information), n represents the index of information V3 (i.e., time domain information), and the value ranges of k and n are both: 0~F; X(k) represents the frequency domain information of the kth timestamp (i.e., information V4), and x(n) represents the time domain information of the nth timestamp (i.e., information V3); e represents a natural constant, j represents an imaginary unit (i.e. j 2 =-1), π represents pi; Step S155: Summarize the information V4 corresponding to the 1st to nth timestamps as the audio to be extracted; Use the SpeechRecognition library and Google Web Speech API to convert the audio to be extracted into text (speech to text) to obtain text β; Combine text α and text β as text information.

[0021] Step S2: extracting color, shape, texture and edge features from the image to be extracted to obtain an image feature set; Calculate the MFCC coefficients of the audio to be extracted, perform homomorphic and cepstrum processing on each voice information in the audio to be extracted, calculate the LPCC coefficients of each voice information, and obtain a voice feature set; See also Figure 2 and Figure 3 , the specific steps of step S2 are as follows: Step S21: using the standard image 1 as image A; Extract color, shape, texture and edge features from image A; Step S211: Obtain pixels of image A, denoted as w×l, where w represents pixel length and l represents pixel width; Use the Pillow library function to get the RGB value of each pixel in image A and get the RGB matrix: ; Among them, X 11 Indicates the RGB value corresponding to the pixel in the first row and first column of image A; X 12 represents the RGB value corresponding to the pixel in the first row and second column of image A; and so on, X wl Indicates the RGB value corresponding to the pixel in the wth row and the lth column in image A; Calculate X 11 ~X wl The average value of is taken as the pixel average value of image A, recorded as aX; calculate X 11 ~X wl The standard deviation of is taken as the pixel standard deviation of image A, denoted as μX; Step S212: Calculate the skewness of each color channel in image A: Calculate the average RGB value of all pixels in the first color channel, denoted as aX(1); the calculation formula of aX(1) is as follows: ; Where j is a positive integer, and the value range of j is: 1~1; X 1j Indicates the RGB value corresponding to the pixel in the 1st row and the jth column in image A; Calculate the skewness of the first color channel, denoted as SS(1); the calculation formula of SS(1) is as follows: ; Calculate the average RGB value of all pixels in the second color channel, denoted as aX(2); the calculation formula of aX(2) is as follows: ; Among them, X 2j Indicates the RGB value corresponding to the pixel in the 2nd row and the jth column in image A; Calculate the skewness of the second color channel, denoted as SS(2); the calculation formula of SS(2) is as follows: ; Similarly, the average RGB value of all pixels in the w-th color channel is calculated, denoted as aX(w); the calculation formula of aX(w) is as follows: ; Among them, X wj Indicates the RGB value corresponding to the pixel in the wth row and the jth column in image A; Calculate the skewness of the w-th color channel, denoted as SS(w); the calculation formula of SS(w) is as follows: ; Calculate the average value of SS(1)~SS(w) as the total skewness of image A, denoted as aSS, and take aX, μX and aSS as the color features of image A; Step S213: In the RGB matrix, read the maximum value of the pixel RGB value and record it as X MAX , the minimum value is recorded as X MIN ; Calculate the average value of pixel RGB value and record it as X AVE ; Perform secondary normalization on image A; Define the calculation process of secondary normalization of each pixel in image A: Let the RGB value of the pixel in the i-th row and j-th column of image A (i.e. RGB matrix) be X ij ;X ij The standardized value of ij ; Judge X ij Is it in X MAX ~X AVE Between, determine CX ij Calculation formula of If X ij In X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1+(X ij -X AVE ) / (X MAX -X AVE )]×128-1; If X ij Not in X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1-(X AVE -X ij ) / (X AVE -X MIN )]×128-1; Step S214: Calculate the standard value of each pixel in the picture A to obtain a binary image of the picture A; Use OpenCV library to extract shape features of binary images; Step S2141: using the Canny edge detection algorithm to find the edge of the binary image; Locate the image based on the edge of the binary image and use the cv2.findContours function to find the contours of each object in the binary image; Step S2142: For the contour of each object in the binary image, use the arcLength() function to calculate the length of the contour; Use the contourArea() function to calculate the area of ​​the contour; Use the boundingRect() function to calculate the bounding box of the contour; Use the approxPolyDP() function to calculate the approximate polygon of the contour; Step S2143: Summarize the contour length, contour area, contour bounding box and contour approximate polygon of each object in the binary image as the shape feature of the image A; Step S215: define a 256×256 empty matrix as an initial gray-level co-occurrence matrix; 1 is the distance between pixels, 0 is o The direction is 256 gray levels. Use the greycomatrix function to scan the binary image and fill in the initial gray co-occurrence matrix to obtain the gray co-occurrence matrix of the image set to be extracted, which is recorded as the co-occurrence matrix A. Count the number of pixel pairs in the co-occurrence matrix A, denoted as pip; use Python and NumPy libraries to calculate the angular second-order moment of each pixel pair in the co-occurrence matrix A and the diagonal mean square difference corresponding to the 1st to pipth pixels as the texture features of image A; Step S22: Repeat the same steps of extracting the color, shape, texture and edge features of the standard image 1 (i.e., step S21) to extract the color, shape, texture and edge features corresponding to the standard images 2 to ts; Summarize the color, shape, texture and edge features corresponding to the standard images 1 to ts as an image feature set; Step S23: Calculate the MFCC coefficients of the audio to be extracted, and perform homomorphic and cepstrum processing on each voice information in the audio to be extracted to obtain the LPCC coefficients of each voice information; Step S231: Obtain the total number F of voice information in the audio to be extracted (i.e., the number of information V4 in the above step S155); The voice information in the audio to be extracted is recorded as information V5; Get the frequency of each message V5, denoted as f 1 、f 2 ~f F ; Among them, f 1 Indicates the frequency of information V5 corresponding to the first timestamp; f 2 Indicates the frequency of information V5 corresponding to the second timestamp; and so on, f F Indicates the frequency of information V5 corresponding to the Fth timestamp; Step S232: Calculate the MFCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S2321: Convert the V5 frequency of the information corresponding to each timestamp into the Mel frequency. The conversion formula of the Mel frequency is as follows: F mel (f i ) = 2595 × lg (1 + f i / 700); where i represents the index of information V5, f i represents the frequency of the information V5 corresponding to the i-th timestamp; F mel (f i ) indicates f in Mel i The perceived frequency (i.e. Mel frequency); Substitute the frequency of the information V5 from the 1st to the Fth timestamp into the Mel frequency conversion formula, calculate the perceived frequency of the 1st to the Fth timestamp, and get F mel (f 1 )~F mel (f F ); Step S2322: performing discrete cosine transform on the perceived frequency of the information V6 to obtain the MFCC coefficients of the audio A to be extracted; The formula for discrete cosine transform is as follows: ; Among them, MFCC (i) represents the MFCC coefficient corresponding to the i-th information V5; F mel (f 1 )~F mel (f F ) is substituted into the discrete cosine transform formula to calculate the MFCC coefficients corresponding to the 1st to Fth timestamps, and MFCC (1) to MFCC (F) are obtained; Step S233: Calculate the LPCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S2331: Perform Z transformation on the 1st to Fth information V5 to obtain Z(1)~Z(F); The function expression of Z transform is as follows: ; where i represents the number of information V5, f i represents the frequency of information V5 corresponding to the i-th timestamp; Z(i) represents the output result of Z-transformation on the i-th information V5; z i The complex variable representing the i-th information V5, z i The calculation formula is as follows: z i = (vc × f i)×e(j×ω) -1 ; where vc represents the speed of sound, e represents a natural constant, and j represents an imaginary unit (i.e. j 2 =-1); ω represents the phase of the i-th information E, and the calculation formula of ω is as follows: ω=arctan[(vc×f i ) / fd]; where arctan represents the inverse tangent function; f 1 ~f F Substitute into the function expression of Z transform, perform Z transform on the 1st to Fth information V5, and obtain Z(1)~Z(F); Step S2332: Perform inverse Z transform on Z(1) to Z(F); The function expression of logarithmic transformation is as follows: ; Wherein, lnZ(i) represents the output result of logarithmic transformation of Z(i); Substitute Z(1)~Z(F) into the function expression of logarithmic transformation, perform logarithmic transformation on the 1st to Fth information V5, and obtain lnZ(1)~lnZ(F); Use the iztrans function to perform inverse Z transform on lnZ(1)~lnZ(F), and transform the information V5 from frequency domain information to time domain information to obtain ts 1 ,ts 2 ~ts F ; Among them, ts 1 Indicates the time domain information of the first timestamp corresponding information V5; ts 2 Indicates the time domain information of the second timestamp corresponding to information V5; and so on, ts F Indicates the time domain information of the information V5 corresponding to the Fth timestamp; Step S2333: ts 1 ~ts F Perform cepstrum processing to obtain the LPC coefficients of the 1st to Fth information E, denoted as ce 1 ~ce F ; The function expression of cepstrum processing is as follows: ; Wherein, i and k both represent the number of information E, and the value range of i and k is: 1~F; f i Indicates the frequency of the information V5 corresponding to the i-th timestamp, ts i Represents the time domain information of the information V5 corresponding to the i-th timestamp; Using the discrete cosine transform algorithm, ce 1 ~ce FConvert the time domain information into frequency domain information to obtain the LPCC coefficients of the audio to be extracted; Step S234: Summarize the MFCC coefficients and LPCC coefficients of the audio to be extracted as a speech feature set of the audio to be extracted.

[0022] Step S3: performing multimodal feature fusion according to the image feature set and the speech feature set to obtain fusion features; using a machine learning algorithm, summarizing and classifying the image to be extracted and the audio to be extracted according to the fusion features, and establishing semantic annotations in combination with text information; The specific steps of step S3 are as follows: Step S31: Summarize the image feature set of each image to be extracted and summarize the voice feature set of the audio to be extracted; Use the image_features object, audio_features object, and text_features object to receive the image feature set, the speech feature set, and the text file in turn to obtain a one-dimensional feature vector; Use NumPy's concatenate function to concatenate the image_features object and the audio_features object to obtain the concatenated_feature object (axis = 1); Use NumPy's concatenate function to concatenate the concatenated_feature object and the audio_features object to obtain the concatenated_features object (axis = 0); Use the concatenated_features object as the fusion feature; Step S32: Use the ResNet-50 model and the ImageNet dataset to summarize and classify the images and audios to be extracted, and establish semantic annotations in combination with text information; The simple implementation code of step S32 is as follows: import torch import torch.nn as nn import torch.optim as optim import torchvision import torchvision.transforms as transforms from torchvision.models import resnet50 from torch.utils.data import DataLoader # Data preprocessing transform = transforms.Compose( [transforms.RandomHorizontalFlip(), # Random horizontal flip transforms.RandomCrop(32, padding=4), # Random cropping transforms.ToTensor(), # Convert to a tensor transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994,0.2010))]) # Normalization # Load the CIFAR-10 dataset trainset = torchvision.datasets.CIFAR10(root='. / data', train=True, download=True, transform=transform) trainloader = DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2) testset = torchvision.datasets.CIFAR10(root='. / data', train=False, download=True, transform=transform) testloader = DataLoader(testset, batch_size=100, shuffle=False, num_workers=2) # Load the ImageNet model model = ImageNet(pretrained=False) # Do not use pre-trained weights num_ftrs = model.fc.in_features model.fc = nn.Linear(num_ftrs, 10) # Modify the final fully connected layer to match the number of categories in CIFAR-10 # Set up the device device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") model = model.to(device) # Loss function and optimizer criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9,weight_decay=1e-4) # Train the model num_epochs = 20 for epoch in range(num_epochs): # multiple rounds of training running_loss = 0.0 for i, data in enumerate(trainloader, 0): inputs, labels = data inputs, labels = inputs.to(device), labels.to(device) # Zero gradient optimizer.zero_grad() # Forward pass outputs = model(inputs) loss = criterion(outputs, labels) Backpropagation and optimization loss.backward() optimizer.step() # Statistical loss running_loss += loss.item() if i % 100 == 99: # print every 100 batches print(f"[{epoch + 1}, {i + 1}] loss: {running_loss / 100:3f}") running_loss = 0.0 print('Finished Training') # Test the model correct = 0 total = 0 with torch.no_grad(): for data in testloader: images, labels = data images, labels = images.to(device), labels.to(device) outputs = model(images) _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() print(f'Accuracy of the network on the 10000 test images: {100 *correct / total} %').

[0023] Step S4: Summarize semantic annotations and provide feedback.

[0024] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technical personnel in this field according to actual conditions. For example, if there are weight coefficients and proportional coefficients, their set sizes are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the size of the weight coefficient and the proportional coefficient, it is sufficient as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0025] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for establishing a semantic index for video surveillance streaming audio and video storage based on multimodality, characterized in that: The method comprises: Step S1: Obtain a surveillance video of a target area; extract a video portion and a voice information portion from the surveillance video to obtain an original video and original voice information; Extract and enhance the frame images in the original video to obtain the image to be extracted; Extract and enhance the audio stream in the original voice information to obtain the audio and text information to be extracted; Step S2: extracting color, shape, texture and edge features from the image to be extracted to obtain an image feature set; Calculate the MFCC coefficients of the audio to be extracted, perform homomorphic and cepstrum processing on each voice information in the audio to be extracted, calculate the LPCC coefficients of each voice information, and obtain a voice feature set; Step S3: performing multimodal feature fusion according to the image feature set and the speech feature set to obtain fusion features; using a machine learning algorithm, summarizing and classifying the image to be extracted and the audio to be extracted according to the fusion features, and establishing semantic annotations in combination with text information; Step S4: Summarize semantic annotations and provide feedback.

2. According to the method of establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 1, it is characterized in that: The specific steps of step S1 are as follows: Step S11: Obtain the surveillance video of the target area and obtain the absolute path of the surveillance video; Step S12: extracting the image part in the surveillance video to obtain the original video; Extract the voice information part in the surveillance video to obtain the original voice information; Step S13: extracting and enhancing the frame image in the original video to obtain an image to be extracted; Get the resolution of the original video, recorded as w×l, where w represents the pixel length of the original video and l represents the pixel width of the original video; Get the number of frames of the original video, recorded as ts; Perform denoising, tilt correction and normalization operations on the first frame image in the original video to obtain a standard image 1; Step S14: Repeat the same step of processing the first frame image in the original video to obtain the standard image 1, and process the second to ts-th frame images in the original video to obtain standard images 2 to standard images ts; Use standard image 1 to standard image ts as images to be extracted; Using OSR technology, scan and extract visible texts in standard images 1 to standard images ts to obtain text α; Step S15: Perform pre-emphasis, windowing and Fourier transform enhancement processing on the original voice information to obtain the audio and text information to be extracted, and then go to step S2.

3. According to the method of establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 2, it is characterized in that: The specific steps of step S13 are as follows: Step S131: record the first frame image in the original video as image A; Let the square root of (w×l) be pa; perform continuous square root operations on (w×l) until the value of pa is between (0,10); pa is a positive integer, rounded down; Step S132: With a convolution kernel size of pa×pa, use the cv2.GaussianBlur function in the OpenCV library to perform Gaussian blur denoising on image A to obtain image A1; Use cv2.cvtHOlor() method to convert image A1 into a grayscale image, and use cv2.threshold() method to convert the grayscale image into a binary image to obtain image A2; Step S133: Use cv2.findHOntours() method to find the largest contour in image A2, use the minimum enclosing rectangle method to calculate the rotation angle of the largest contour, use cv2.getRotationMatrix2D and cv2.warpAffine function to rotate image A2 to obtain image A3; Step S134: Read the grayscale value of each pixel in image A3, denoted as HO （1,1） , HO （1,2） ~HO （w,l） ; Among them, HO （1,1） Indicates the gray value corresponding to the pixel at (1,1) in image A3; HO （1,2） Indicates the gray value corresponding to the pixel at (1,2) in image A3; and so on, HO （w,l） Represents the grayscale value corresponding to the pixel at (w, l) in image A3; Step S135: extract the maximum value of the pixel grayscale in image A3 and record it as R MAX , the minimum value is denoted as R MIN ; Calculate the average grayscale value of all pixels in image A3, denoted as R AVE ; Normalize the pixel at (1,1) in image A3 and calculate HO （1,1） The standardized value of （1,1） ; Determine HO （1,1） Is it in R MAX ~R AVE between; If yes, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(HO （1,1） -R AVE ) / (R MAX -R AVE )]×256} / 2-1;CB （1,1） Round up; If not, then CB （1,1） The calculation formula is as follows: CB （1,1） ={[(R AVE -HO （1,1） ) / (R AVE -R MIN )]×256} / 2-1;CB （1,1） Round down; Step S136: Repeat calculation of HO （1,1） The same steps as the normalized values ​​are used to calculate HO （1,2） ~HO （w,l） The standardized value of is used to complete the standardization of image A3 and obtain the standard image 1.

4. According to the method of establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 2, it is characterized in that: The specific steps of step S15 are as follows: Step S151: obtaining the duration of the original voice information, denoted as vd; Calculate the rated number of samples of the original speech information, denoted as sq; sq calculation formula: sq = [log 60 (vd)] × 1024; Calculate the frame duration of a single voice message, recorded as fd; Calculation formula of fd: fd = vd we / sq; Calculate the frame shift duration of a single voice message, denoted as fsd; The calculation formula of fsd: fsd = fd / 2; Step S152: Record the original voice information as information V; Use a high-pass filter to remove low-frequency noise in information V to obtain information V1; The information V1 is framed by the frame length fd and the frame shift length fsd to obtain the information V2; The amount of information V2 is recorded as F, F = vd / fd; F is a positive number, rounded up; Step S153: Using a Hamming window to perform windowing processing on the information V2 to obtain information V3; the number of information V3 is the same as the number of information V2; The function expression w(n) of the Hamming window is as follows: w(n)=0.54-0.46cos[[2π×(n-1)] / F)]; wherein n is a positive integer, n represents the index of the nth information V2, and the value range of n is: 0-F; w(n) represents the function expression for converting the nth information V2 into the nth information V3; Step S154: using Fourier transform, converting the information V3 of each timestamp into the frequency domain information of the corresponding timestamp to obtain information V4; the number of information V4 is the same as the number of information V3; The functional expression of Fourier transform is as follows: ； Wherein, k represents the index of information V4, n represents the index of information V3, and the value range of k and n is: 0~F; X(k) represents the frequency domain information of the kth timestamp, and x(n) represents the time domain information of the nth timestamp; e represents a natural constant, j represents an imaginary unit, and π represents the ratio of circumference to circumference; Step S155: Summarize the information V4 corresponding to the 1st to nth timestamps as the audio to be extracted; Use the SpeechRecognition library and Google Web Speech API to convert the audio to be extracted into text to obtain text β; Combine text α and text β as text information.

5. According to claim 2, a method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality is characterized in that: The specific steps of step S2 are as follows: Step S21: using the standard image 1 as image A; Extract color, shape, texture and edge features from image A; Step S22: Repeat the same steps of extracting the color, shape, texture and edge features of the standard image 1 to extract the color, shape, texture and edge features corresponding to the standard images 2 to ts; Summarize the color, shape, texture and edge features corresponding to the standard images 1 to ts as an image feature set; Step S23: Calculate the MFCC coefficients of the audio to be extracted, and perform homomorphic and cepstrum processing on each voice information in the audio to be extracted to obtain the LPCC coefficients of each voice information; Step S231: Obtain the total amount F of voice information in the audio to be extracted; The voice information in the audio to be extracted is recorded as information V5; Get the frequency of each message V5, recorded as f1, f2~f F ; Among them, f1 represents the frequency of information V5 corresponding to the first timestamp; f2 represents the frequency of information V5 corresponding to the second timestamp; and so on. F Indicates the frequency of information V5 corresponding to the Fth timestamp; Step S232: Calculate the MFCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S233: Calculate the LPCC coefficients corresponding to the 1st to Fth timestamps according to the information V5; Step S234: Summarize the MFCC coefficients and LPCC coefficients of the audio to be extracted as the speech feature set of the audio to be extracted, and proceed to step S3.

6. The method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 5 is characterized in that: The specific steps of step S21 are as follows: Step S211: Obtain pixels of image A, denoted as w×l, where w represents pixel length and l represents pixel width; Use the Pillow library function to get the RGB value of each pixel in image A and get the RGB matrix: ; Among them, X 11 Indicates the RGB value corresponding to the pixel in the first row and first column of image A; X 12 represents the RGB value corresponding to the pixel in the first row and second column of image A; and so on, X wl Indicates the RGB value corresponding to the pixel in the wth row and the lth column in image A; Calculate X 11 ~X wl The average value of is taken as the pixel average value of image A, recorded as aX; calculate X 11 ~X wl The standard deviation of is taken as the pixel standard deviation of image A, denoted as μX; Step S212: Calculate the skewness of each color channel in image A: Calculate the average RGB value of all pixels in the first color channel, denoted as aX(1); the calculation formula of aX(1) is as follows: ; Where j is a positive integer, and the value range of j is: 1~1; X 1j Indicates the RGB value corresponding to the pixel in the 1st row and the jth column in image A; Calculate the skewness of the first color channel, denoted as SS(1); the calculation formula of SS(1) is as follows: ； Calculate the average RGB value of all pixels in the second color channel, denoted as aX(2); the calculation formula of aX(2) is as follows: ; Among them, X 2j Indicates the RGB value corresponding to the pixel in the 2nd row and the jth column in image A; Calculate the skewness of the second color channel, denoted as SS(2); the calculation formula of SS(2) is as follows: ； Similarly, the average RGB value of all pixels in the w-th color channel is calculated, denoted as aX(w); the calculation formula of aX(w) is as follows: ; Among them, X wj Indicates the RGB value corresponding to the pixel in the wth row and the jth column in image A; Calculate the skewness of the w-th color channel, denoted as SS(w); the calculation formula of SS(w) is as follows: ； Calculate the average value of SS(1)~SS(w) as the total skewness of image A, denoted as aSS, and take aX, μX and aSS as the color features of image A; Step S213: In the RGB matrix, read the maximum value of the pixel RGB value and record it as X MAX , the minimum value is recorded as X MIN ; Calculate the average value of pixel RGB value and record it as X AVE ; Perform secondary normalization on image A; Define the calculation process of secondary normalization of each pixel in image A: Let the RGB value of the pixel in the i-th row and j-th column of image A be X ij ;X ij The standardized value of ij ; Judge X ij Is it in X MAX ~X AVE Between, determine CX ij Calculation formula of If X ij In X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1+(X ij -X AVE ) / (X MAX -X AVE )]×128-1; If X ij Not in X MAX ~X AVE Between, then CX ij The calculation formula is as follows: CX ij =[1-(X AVE -X ij ) / (X AVE -X MIN )]×128-1; Step S214: Calculate the standard value of each pixel in the picture A to obtain a binary image of the picture A; Use OpenCV library to extract shape features of binary images; Step S215: define a 256×256 empty matrix as an initial gray-level co-occurrence matrix; 1 is the distance between pixels, 0 is o The direction is 256 gray levels. Use the greycomatrix function to scan the binary image and fill in the initial gray co-occurrence matrix to obtain the gray co-occurrence matrix of the image set to be extracted, which is recorded as the co-occurrence matrix A. The number of pixel pairs in the co-occurrence matrix A is counted, denoted as pip; the angular second-order moment of each pixel pair in the co-occurrence matrix A and the diagonal mean square difference corresponding to the 1st to pipth pixels are calculated using Python and NumPy libraries as the texture features of image A.

7. The method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 5 is characterized in that: The specific steps of step S233 are as follows: Step S2331: Perform Z transformation on the 1st to Fth information V5 to obtain Z(1)~Z(F); The function expression of Z transform is as follows: ; where i represents the number of information V5, f i represents the frequency of information V5 corresponding to the i-th timestamp; Z(i) represents the output result of Z-transformation on the i-th information V5; z i The complex variable representing the i-th information V5, z i The calculation formula is as follows: z i = (vc × f i )×e(j×ω) -1 ; Wherein, vc represents the speed of sound, e represents the natural constant, j represents the imaginary unit, ω represents the phase of the i-th information E, and the calculation formula of ω is as follows: ω=arctan[(vc×f i ) / fd]; f1~f F Substitute into the function expression of Z transform, perform Z transform on the 1st to Fth information V5, and obtain Z(1)~Z(F); Step S2332: Perform inverse Z transform on Z(1) to Z(F); The function expression of logarithmic transformation is as follows: ; Wherein, lnZ(i) represents the output result of logarithmic transformation of Z(i); Substitute Z(1)~Z(F) into the function expression of logarithmic transformation, perform logarithmic transformation on the 1st to Fth information V5, and obtain lnZ(1)~lnZ(F); Use the iztrans function to perform inverse Z transform on lnZ(1)~lnZ(F), transform the information V5 from frequency domain information to time domain information, and obtain ts1, ts2~ts F ; Among them, ts1 represents the time domain information of the information V5 corresponding to the first timestamp; ts2 represents the time domain information of the information V5 corresponding to the second timestamp; and so on, ts F Indicates the time domain information of the information V5 corresponding to the Fth timestamp; Step S2333: ts1 to ts F Perform cepstrum processing to obtain the LPC coefficients of the 1st to Fth information E, denoted as ce1~ce F ; The function expression of cepstrum processing is as follows: ; Wherein, i and k both represent the number of information E, and the value range of i and k is: 1~F; f i Indicates the frequency of the information V5 corresponding to the i-th timestamp, ts i Represents the time domain information of the information V5 corresponding to the i-th timestamp; Using the discrete cosine transform algorithm, ce1~ce F Convert the time domain information into frequency domain information to obtain the LPCC coefficients of the audio to be extracted.

8. The method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 5 is characterized in that: The specific steps of step S3 are as follows: Step S31: Summarize the image feature set of each image to be extracted and summarize the voice feature set of the audio to be extracted; Use the image_features object, audio_features object, and text_features object to receive the image feature set, the speech feature set, and the text file in turn to obtain a one-dimensional feature vector; Use NumPy's concatenate function to concatenate the image_features object and the audio_features object to get the concatenated_feature object; Use NumPy's concatenate function to concatenate the concatenated_feature object and the audio_features object to obtain the concatenated_features object; Use the concatenated_features object as the fusion feature; Step S32: Use the ResNet-50 model and the ImageNet dataset to summarize and classify the images and audios to be extracted, and establish semantic annotations in combination with text information.

9. The method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 5 is characterized in that: The specific steps of step S232 are as follows: Step S2321: Convert the V5 frequency of the information corresponding to each timestamp into the Mel frequency. The conversion formula of the Mel frequency is as follows: F mel (f i ) = 2595 × lg (1 + f i / 700); where i represents the index of information V5, f i represents the frequency of the information V5 corresponding to the i-th timestamp; F mel (f i ) indicates f in Mel i The perceived frequency; Substitute the frequency of the information V5 from the 1st to the Fth timestamp into the Mel frequency conversion formula, calculate the perceived frequency of the 1st to the Fth timestamp, and get F mel (f1)~F mel (f F ); Step S2322: performing discrete cosine transform on the perceived frequency of the information V6 to obtain the MFCC coefficients of the audio A to be extracted; The formula for discrete cosine transform is as follows: ； Among them, MFCC (i) represents the MFCC coefficient corresponding to the i-th information V5; F mel (f1)~F mel (f F ) is substituted into the discrete cosine transform formula to calculate the MFCC coefficients corresponding to the 1st to Fth timestamps, and MFCC (1) to MFCC (F) are obtained.

10. The method for establishing a video surveillance streaming audio and video storage semantic index based on multimodality according to claim 6, characterized in that: The specific steps of step S214 are as follows: Step S2141: using the Canny edge detection algorithm to find the edge of the binary image; Locate the image based on the edge of the binary image and use the cv2.findContours function to find the contours of each object in the binary image; Step S2142: For the contour of each object in the binary image, use the arcLength() function to calculate the length of the contour; Use the contourArea() function to calculate the area of ​​the contour; Use the boundingRect() function to calculate the bounding box of the contour; Use the approxPolyDP() function to calculate the approximate polygon of the contour; Step S2143: Summarize the contour length, contour area, contour bounding box and contour approximate polygon of each object in the binary image as the shape feature of image A.

Images