Structured acquisition method for edge video stream image data

Abstract

The invention discloses a structured acquisition method for edge video stream image data, and the method comprises the steps: collecting video stream image data, inputting the video stream image data into an edge calculation platform, and carrying out the segmentation and preprocessing of a video stream, and obtaining the motion range of a target; constructing a target tracking method according to the motion range of the target to obtain time sequence data of the target; according to the time sequence data of the target, constructing a target automatic labeling model, training the automatic labeling model, and updating parameters corresponding to the model; associating annotation information generated by the automatic annotation model after parameter updating with an image, establishing a corresponding relationship of data, and generating structured data with an association relationship; storing the structured data in a database, and adding an adaptive index to optimize query; according to the characteristics of an edge computing platform, the video stream image data is efficiently and accurately acquired, and the video stream data is applied to subsequent target tracking and video content retrieval after being subjected to structured processing.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a structured acquisition method for edge video stream image data. Background Art

[0002] In the field of computer vision, methods for collecting video stream image data are crucial for accurately analyzing and understanding the characteristics of dynamic scenes. Video stream image data is temporal and dynamic, so collecting data containing timestamp information and contextual semantics is crucial for analyzing video stream image data.

[0003] With the rapid development of information technology and the popularity of big data, data storage issues have become very important. Compared with raw data, structured data uses a more compact representation, which can store and manage data more efficiently. At the same time, structured data facilitates the establishment of databases and indexes for fast retrieval and query.

[0004] Edge computing is a distributed computing model that deploys data processing and storage capabilities at the edge of the network, close to the data source or user. Its core goal is to reduce reliance on remote cloud computing and improve efficiency through localized processing. However, when applying traditional methods for collecting video stream image data to edge computing, problems such as poor real-time performance and data inconsistency are observed. Furthermore, image quality is affected by a variety of factors, and the data collected by traditional methods may contain blur, noise, and distortion. Furthermore, traditional methods typically focus only on the image itself and lack contextual semantic information, which limits higher-level understanding and analysis of video data.

[0005] Multi-head attention networks are a key component in deep learning. They improve the model's expressiveness by computing multiple heads in parallel, learning different feature representation subspaces and extracting richer and more informative feature representations. This mechanism gives multi-head attention networks unique advantages when processing video stream image data. Therefore, combining multi-head attention networks with edge computing to achieve structured acquisition of edge-oriented video stream image data should overcome the shortcomings of existing technologies. Summary of the Invention

[0006] The purpose of this invention is to provide a structured acquisition method for edge video stream image data, which uses image data processing methods and multi-head attention networks to efficiently acquire and store structured video stream image data.

[0007] A structured acquisition method and system for edge video stream image data, comprising the following steps:

[0008] S1. Collect video stream image data and input it into the edge computing platform, segment and preprocess the video stream to obtain the target's motion range;

[0009] S2. Build a target tracking method based on the target's motion range to obtain the target's time series data;

[0010] S3. Build an automatic image target annotation model based on the target’s time series data, train the automatic annotation model, and update the corresponding model parameters;

[0011] S4, associating the annotation information generated by the automatic annotation model after the updated parameters with the image, establishing a corresponding relationship between the data, and generating structured data with an associated relationship;

[0012] S5. Store structured data in the database and add adaptive indexes for query optimization.

[0013] Furthermore, the step S1 specifically includes:

[0014] S101, collecting video image data and inputting it into the edge computing platform;

[0015] S102, storing the collected data in the form of a video sequence;

[0016] S103, intercepting t frames of image from the continuous image frames each time for processing to reduce the amount of calculation and storage requirements;

[0017] S104: Input the captured images in sequence and perform grayscale processing on the images. The grayscale value Grey calculation formula of the images is as follows:

[0018] Grey=0.299R+0.587G+0.114B

[0019] S105: Divide the target's motion range by extracting a background template.

[0020] Furthermore, the specific steps in step S105 are as follows:

[0021] The first input frame X1 is selected as the reference frame, and the vector x1(i, j) represents the grayscale value of the pixel (i, j) of the first frame image, where i = 1, 2, 3, ..., M, j = 1, 2, 3, ..., N, M represents the width of the input video frame, N represents the height of the input video frame, and ... t (i, j) represents the grayscale value of the pixel (i, j) in the t-th frame image, t is a positive integer greater than or equal to 2, and t consecutive video frames are represented by the vector set [x1, x2, x3, ..., x t ] is used to represent the gray value difference f between the t-th frame image and the reference frame at pixel (i, j) 1t The calculation formula is as follows:

[0022] f 1t＝ ||x t (i,j)|-|x1(i,j)||

[0023] The above formula is used to compare the grayscale value difference of the continuous t-frame image with the reference frame at the pixel (i, j). The largest value F is selected from them to determine whether the grayscale value of the pixel has changed, thereby determining the area of this pixel. The F value calculation formula is as follows:

[0024] F=max{f 12 ,f 13 ,…,f 1t}

[0025] According to the above formula, the maximum grayscale difference value of all pixels is stored in the matrix S M×N When F is 0, it indicates that the pixel has not changed and is stored in S with a value of 0; when F is not equal to 0, let Max t i,j and Min t i,j They are the maximum and minimum grayscale values of the pixel point in the t-th frame compared with the previous t-1 frame; set the threshold T, when Max t i,j and Min t i,j When the difference value corresponding to the pixel point is less than the threshold, it is classified as the background area. i,j Set to 0;

[0026]

[0027] Among them, S i,j Represents the binary image at pixel (i, j), if S i,j If the value of is 1, then this pixel does not belong to the background area, and the pixels with a value of 1 are connected to generate a motion area.

[0028] Furthermore, the step S2 specifically includes:

[0029] S201, performing noise reduction processing on the divided motion areas;

[0030] S202, setting a filter window of size n×n, taking the strong edge point (i, j) as the center, using detection operators in eight directions to calculate the gradient amplitude, and obtaining the target strong edge information;

[0031] S203, retaining edge details of the local gradient to obtain the edge structure of the target to be identified in the image;

[0032] S204, using the maximum inter-class variance method to connect the edges and form a complete target boundary;

[0033] S205, constructing a video stream image target tracking method based on the complete target boundary and the divided motion area;

[0034] Furthermore, the step S201 specifically includes:

[0035] Use the filter window to filter the noise points in the image data. First, filter the suspected noise points in the image, calculate the grayscale average of each point in the window, and set the threshold T1 to filter the suspected noise. The formula of T1 is defined as follows:

[0036]

[0037] Among them, m, n represent the length and width of the filter window, h(i,j), Represents the grayscale value of pixel (i, j) and the mean grayscale value of the four area windows above, below, left and right respectively; when the condition h is not met max -T1 <h(i,j)<h max and h min <h(i,j)<h min At +T1, the pixel is a suspected noise point;

[0038] After filtering out the suspected noise points, When , this point is a noise point, and the calculation formula of the threshold T2 is as follows:

[0039]

[0040] Where P represents the number of noise points in the current window;

[0041] After obtaining the noise point, the image is denoised according to the number of noise points around the noise point. The noise point is denoised and its grayscale value is replaced. The formula is as follows:

[0042]

[0043] Among them, h represents the gray value of the pixel, h m is the gray value of the non-noise point in the window.

[0044] Furthermore, the step S202 specifically includes:

[0045] In order to obtain more edge information, an n×n operator template is designed to select the detection operators in eight directions of 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, with the strong edge point (i, j) as the center.

[0046] The calculation formula of the gradient amplitude is:

[0047]

[0048] The gradient direction is arctan(G 90 / G0), where the subscript 0 of G0 represents the 0 degree direction, G 90 The subscript 90 indicates the 90-degree direction, and the same applies to the others.

[0049] Furthermore, the step S203 specifically includes:

[0050] Among them, the adjacent pixels in the gradient direction of each pixel are checked, and only the local maximum values of the edge are retained;

[0051]

[0052] Where δ represents the mediation factor that can modify the degree of edge refinement;

[0053] Furthermore, the step S204 specifically includes:

[0054] First, the grayscale value ranges of the background area and the motion area are defined as two categories, namely: B1 = [0, k], B2 = [k, b], and the grayscale value range of the entire image is [0, b]. Then, the proportion of the number of pixels with grayscale values of the two categories relative to the global number of pixels is calculated, and the calculated result is multiplied by the average grayscale value of each category to obtain the inter-class variance formula Δ(k). The maximum inter-class variance is calculated iteratively to obtain the optimal segmentation threshold k. The inter-class variance calculation formula is as follows:

[0055]

[0056] B2=1-B1

[0057] Δ(k)=B1×b1 2 +B2×b2 2

[0058] Where b1 and b2 represent the average grayscale of B1 and B2 classes respectively;

[0059] Furthermore, the step S205 specifically includes:

[0060] After obtaining the complete boundary and motion area information of the target, the target object r0 is selected and d background images r around the target are extracted from its adjacent areas. i As a regularization term, it is introduced into the ridge regression equation, the context information is superimposed on the target area, and the target area is modeled. The calculation is as follows:

[0061]

[0062] Where g() represents the objective function, γ is the ridge coefficient, σ is the learned correlation filter, R represents the feature matrix of the target and the extracted surrounding background image, Represents the relationship matrix between the target and the surrounding background image, I n represents the identity matrix representation, represents the square of L2 norm;

[0063] The target response image is constructed based on the learned correlation filter, and the maximum response position detection is used to determine the position of the target in the current image. The detection formula is:

[0064]

[0065] In the formula, τ represents the regularization parameter to prevent overfitting, g w represents the surrounding background search window, ⊙ represents the point-by-point product operation;

[0066] According to the target results obtained by detection, two levels of target features are divided, and the deep features of the target position are defined as l d , the shallow features of the target details are l s , construct a weighted combination of features;

[0067] L(r)=λ1×l d (r)+λ2×l s (r)

[0068] λ1+λ2=1

[0069] Where r represents the target object, λ1 and λ2 represent the weights of deep features and shallow features respectively, and L represents the score value at the target position. After selecting the target, other frame entities are selected for comparison with the target entity, and the score value L(r*) of the candidate target is calculated. The predicted target is selected based on the difference in the score between the candidate target and the target. The minimization loss function Loss is defined as:

[0070]

[0071] In the formula, κ is the rate of change of the distance between two entities, θ() is a function that penalizes excessively large weight values, and e () represents the exponential function with base e;

[0072] Construct the time series data of the target and store the time and location information of the target during the tracking process in the form of time series data;

[0073] The model is updated based on the fluctuation of the response graph to ensure the continuity of the tracking process. When the target is disturbed, the response graph will fluctuate violently; when the target is not disturbed, the peak of the response graph is clearer. The dynamic evaluation index APCE is used to determine whether the model needs to be updated.

[0074]

[0075] Where y max Indicates the peak value of response, y min Represents the minimum response value, (w,h) represents the position of the target, and mean() represents the mean function.

[0076] Furthermore, the step S3 specifically includes:

[0077] S301, extracting the target's features in time and space dimensions based on the tracking results;

[0078] S302: construct an automatic target labeling model based on the target characteristics, and construct a loss function of the model based on the automatic target labeling model and the training data;

[0079] S303. Calculate the gradient of each parameter in the model according to the loss function, and perform backpropagation to update the parameters in the model.

[0080] Furthermore, the step S301 specifically includes:

[0081] First, based on the collected time series data, the position sequence of the surrounding entities and the target entity in the search area at time t is calculated. Encode, generate embedding vectors, and encode the time steps. The formula is as follows:

[0082]

[0083] Among them, e represents the parity of the time step, v represents the dimension of the timestamp feature; the position sequence W is added to the time step sequence Encode(t) to obtain the embedded sequence within time t

[0084] In addition, by increasing the association relationship between the target entity and the surrounding entities at different time steps, context information and scene semantics can be obtained; for each time step t, the trajectory points of different entities are set to W t Input to the multi-head attention network, each multi-head attention network contains two multi-head attention layers, each multi-head attention layer uses G L Each subgraph attention layer focuses on the association relationship of the target entity in different subspaces. The association feature formula Z of the entity at each time step t is t as follows:

[0085]

[0086] Among them, μ is the shared parameter of the entity node, Entity represents the target entity, LeakyReLU() represents the activation function with leakage parameters, and [·||·] represents the concat connection operation of the result after the node is transformed. Indicates mapping the concatenated high-dimensional features into a real number. Indicates G L The related features of the sub-graph attention layers are fused;

[0087] Combine the sequence correlation features of all time steps into Perform embedding vector encoding on Z to generate spatial sequence Z w ;

[0088] Furthermore, the step S302 of constructing the target automatic labeling algorithm model specifically includes:

[0089] The time series Y of the target entity is linearly divided into three vectors Q, K, and V, which are input into the multi-head self-attention layer to obtain the correlation of the target at different time steps. The output results are residually added and passed through the corresponding normalization layer to prevent the gradient from disappearing. The two layers of fully connected layers FCL are connected to output the kinematic feature Y of the target object. g ;

[0090] Attention1=Norm(Y+MultiHead-SelfAtt1(Q1,K1,V1;ψ1))

[0091]

[0092]

[0093] Y g =Norm(Attention1+FCL 2,out )

[0094] In the formula, ψ is the relevant parameter, ReLU() is the activation function, and Norm() represents the normalization function;

[0095] The associated feature Z of the target entity at all time steps w Perform linear differentiation to generate three matrices Q1, K1, and V1. After the following operations, the entity interaction feature Z is obtained g ;

[0096] Z g =Norm(Attention2+FCL 2,out )

[0097] Attention2=Norm(MultiHead-SelfAtt2(Q1,K1,V1;ψ2)+Z w )

[0098]

[0099]

[0100] Reasonably assign weights to motion features and interaction features to generate a comprehensive feature vector η, which is calculated as follows:

[0101] η=β1⊙Y g +β2⊙Z g

[0102]

[0103] β2=1-β1

[0104] in, Represents motion feature Y g The corresponding weight parameters, represents the corresponding bias parameter, ⊙ represents the point-by-point product operation;

[0105] The vector η is input into the fully connected layer for target classification and regression operations to determine the category of the target; the loss function calculation formula Loss1 is as follows:

[0106]

[0107] Among them, p i represents the true category of the target, p j Represents the predicted category probability; sets the minimum bounding rectangle of the annotation box according to the position and size of the target, sets the annotation and color of the annotation box according to the category of the target, uses the same annotation box color for the same category, and stores the annotation information in the annotation dataset; ln() represents the logarithmic function; N represents the number of target objects.

[0108] Furthermore, the step S4 specifically includes:

[0109] S401, extracting annotation information and generating a unique identifier;

[0110] S402, associating the annotation information with the unique identifier with the image dataset to ensure that each image has corresponding annotation information and establish a data correspondence relationship;

[0111] Among them, the image features are added to the annotation information, and a mapping function Φ1 is established to generate a new annotation information dataset Θ;

[0112] Ф1(messagei )=[c1 1 (message i ),c2 1 (message i ),…,c u 1 (message i ),

[0113] c1 2 (message i ),c2 2 (message i ),…,c u 2 (message i ),

[0114] …

[0115] c1 l (message i ),c2 l (message i ),…,c u l (message i )]

[0116] Among them, message i Represents the labeled information, i∈(1,u) represents the number of labeled information, and l represents the dimension;

[0117] S403: Integrate all images and corresponding annotation information according to the established data correspondence to generate structured data with correlation;

[0118] Among them, according to the correlation relationship in the labeled data, the correlation relationship matrix V, v ij is an element of V;

[0119]

[0120] Where, Respectively represent c i and c j The average value of and Respectively represent c i and c j The standard deviation of V is established; a mapping function Ф2 is established to fuse V with the new annotation information dataset Θ to generate a structured dataset with an associated relationship.

[0121] Furthermore, the step S5 specifically includes:

[0122] S501, storing the structured data in a database according to the association relationship;

[0123] S502, extracting the key value of the key-value data into a data element form including the object word key.obj, the qualifier key.lim, and the property word key.par;

[0124] Data element Key = {key.obj}∪{key.lim}∪{key.par};

[0125] S503. When data elements have the same key value but different key values, add the data item KE with the highest discrimination among the key values after the data element at that time; when the index positions are the same, store the corresponding data elements in the form of a linked list; and add a fine-grained locking mechanism in each linked list;

[0126] Among them, the data items are matched for similarity, and the comparison algorithm for obtaining the data item with the greatest discrimination in the key value is:

[0127]

[0128] Where Price() represents the cost parameter of converting two data items into the same data item, and length() represents the length of the data item;

[0129] S504. When querying data, first determine whether the data element can be queried within the specified number of times; if the data element exists, find the key value corresponding to the data element; finally, verify whether the key value meets the requirements. If the key value is consistent, return the key value corresponding to the key value.

[0130] The beneficial effects of the present invention include:

[0131] (1) This invention takes into account the characteristics of the edge computing platform itself and, based on the idea of video stream data processing, first intercepts continuous image frames of video data from multiple sensors as input each time. This not only reduces the computational complexity of image processing and preserves the sequential relationship and temporal information between image frames, but also provides more comprehensive and accurate information for subsequent target detection and modeling. In addition, extracting the background template can make the position of the moving target more prominent, reduce false alarms that may be caused by interference factors, and improve the accuracy of target detection;

[0132] (2) The present invention fully integrates the background information of the moving target to perceive and recognize contextual semantic information, effectively reducing the interference of false boundaries, and can better correct the erroneous tracking caused by target morphological changes or other factors during the target tracking stage;

[0133] (3) The present invention proposes an automatic labeling model that can analyze the motion characteristics of moving targets and their interaction characteristics with other targets, make full use of context information, classify based on feature fusion, and automatically label according to the classification results, thereby improving the accuracy of the model;

[0134] (4) The present invention stores the time and target position information in the form of time series data during the target tracking process based on video stream data, which can retain the position change information of the target position at different time points and dynamically update it according to the fluctuation degree of the response graph. It can adapt to the target changes in different scenarios and has the ability to resist interference;

[0135] (5) The present invention utilizes a multi-head attention network to fully explore the spatial and temporal relationships of targets in video stream image data, and is able to focus on different feature subspaces and capture key information of targets in multiple dimensions, thereby making the model recognition more comprehensive. At the same time, the multi-head attention network can adaptively allocate attention, focusing on key time periods and spatial regions, and better capturing the dynamic changes of targets.

[0136] (6) The present invention improves the traditional indexing method and adopts an adaptive indexing algorithm to optimize data query operations. It uses data elements to replace traditional key values. Data elements contain object words, qualifiers, and characteristic words, which can more accurately represent the characteristics and content of data. When the data elements are the same, the data items and data elements are compared based on their distinguishability, and the corresponding data elements are stored in the form of linked lists. At the same time, a fine-grained locking mechanism is added to each linked list to ensure data security and consistency.

[0137] (7) The present invention automatically annotates the collected video stream data, performs dimensionality-upgrading processing on the data, associates the annotation information with a unique identifier with the image data set, ensures that each image has corresponding annotation information, establishes a data correspondence, and integrates all images and corresponding annotation information based on the established data correspondence to generate structured data with an association relationship, thereby introducing more data feature attributes, making the data more comprehensive and rich, effectively improving the data representation capability, and expanding the application scenarios and fields of the data.

[0138] Other advantages, objectives, and features of the present invention will be elaborated in detail in the following description. Those skilled in the art will clearly appreciate these advantages and features through further study of the following description and gain valuable insights from the practice of the present invention. The objectives and other advantages of the present invention are realized and embodied in the following description and the aforementioned claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0139] Figure 1A schematic diagram of a structured acquisition method and system flow for edge video stream image data of the present invention;

[0140] Figure 2 Converting a binary image acquired through data collection according to an embodiment of the present invention into an example image of a motion region;

[0141] Figure 3 This is an example diagram of target detection according to an embodiment of the present invention;

[0142] Figure 4 This is an example diagram of the gradient calculation template of the present invention;

[0143] Figure 5 A schematic diagram of the structure of the automatic annotation model for the present invention;

[0144] Figure 6 This is an example diagram of the annotation information obtained by the automatic target annotation model according to an embodiment of the present invention;

[0145] Figure 7 A schematic diagram of forming structured data according to an embodiment of the present invention;

[0146] Figure 8 Flowchart of using index to query structured data for the present invention. DETAILED DESCRIPTION

[0147] The preferred embodiments of the present invention are described in detail below. It should be understood that the preferred embodiments are only for illustrating the present invention and are not intended to limit the scope of protection of the present invention.

[0148] like Figure 1 As shown, a structured acquisition method for edge video stream image data of the present invention includes the following steps:

[0149] S1. Collect video stream image data and input it into the edge computing platform. The video stream is segmented and preprocessed to obtain the target's motion range.

[0150] S2. Build a target tracking method based on the target's motion range to obtain the target's time series data;

[0151] S3. Build an automatic image target annotation model based on the target’s time series data, train the automatic annotation model, and update the corresponding model parameters;

[0152] S4, associating the annotation information generated by the automatic annotation model after the updated parameters with the image, establishing a corresponding relationship between the data, and generating structured data with an associated relationship;

[0153] S5. Store the structured data in the database and add an adaptive keyword index for query optimization.

[0154] It should be noted that any process or method description in the flowchart of the present invention or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing steps of a specific logical function or process, and that the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0155] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0156] like Figures 2 to 8 As shown, the specific steps of the above method will be further described below through a specific embodiment.

[0157] The step S1 includes the following sub-steps:

[0158] Step S101: Collect video image data and input it into the edge computing platform. "Collect" herein includes but is not limited to collecting data using tools such as cameras and webcams;

[0159] The video image data collected in step S101 includes the following data information: {video frame: D frame , pixel information: D pixel , color information: Dcolor , image resolution: D resolution , frame rate: D rate , compression encoding format: D encoding}, the acquired image width D weight is 400 pixels, height D height 320 pixels;

[0160] Step S102: storing the collected data in the form of a video sequence;

[0161] Create video sequence D video , loop read the video frame and correspond the pixel information and color information to each frame X i , write each frame to D video middle;

[0162] Step S103: intercepting t frames of image from the continuous image frames each time for processing to reduce the amount of calculation and storage requirements;

[0163] Step S104: Input the captured images in sequence and perform grayscale processing on the images. The grayscale value Grey of the images is calculated using the following formula:

[0164] Grey=0.299R+0.587G+0.114B

[0165] Where R, G, and B are the color values of the red, green, and blue channels corresponding to the pixel, respectively, and the value range is 0 to 255;

[0166] Step S105: Divide the target's motion range by extracting the background template;

[0167] In step 105, the specific process of the method for dividing the motion range is as follows:

[0168] (1) Select the first input frame X1 as the reference frame, take the point in the lower left corner as the origin, the lower boundary of the image as the x-axis, and the left boundary of the image as the y-axis, and construct the coordinates (i, j) of the pixel position;

[0169] (2) Let vector x1(i, j) represent the grayscale value of pixel (i, j) in the first frame image, where i = 1, 2, 3, ..., D weight ,j=1,2,3,…,D height ; Let x t (i, j) represents the grayscale value of pixel (i, j) in the t-th frame, t is a positive integer greater than or equal to 2, and t consecutive video frames can be represented by the vector set [x1, x2, x3, ..., x t ] to indicate;

[0170] (3) The grayscale value difference f between the t-th frame image and the reference frame at pixel (i, j)1t The calculation formula is as follows:

[0171] f 1t＝ ||x t (i,j)|-|x1(i,j)||

[0172] (4) Compare the grayscale value difference of the continuous t-frame image with the grayscale value of the reference frame at the pixel (i, j), select the largest value F, and judge whether the grayscale value of the pixel has changed, so as to determine the area of the pixel. The F value calculation formula is as follows:

[0173] F=max{f 12 ,f 13 ,…,f 1t}

[0174] (5) According to the above formula, the maximum grayscale difference value of all pixels is stored in the matrix S Dweight×Dheight When F is 0, it indicates that the pixel has not changed and is stored in S with a value of 0; when F is not equal to 0, the grayscale value of the pixel may change due to other factors but still belongs to the background area, so Max t i,j and Min t i,j They are the maximum and minimum grayscale values of the pixel point in the t-th frame compared with the previous t-1 frame; set the threshold T, when Max t i,j and Min t i,j When the difference value corresponding to the pixel point is less than the threshold, it is classified as the background area. i,j Set to 0, otherwise it belongs to the motion area, and S i,j Set to 1;

[0175]

[0176] (6) According to the background binary image, the moving target area is roughly divided, such as Figure 2 As shown; traverse the S matrix until a pixel (i, j) with a value of 1 is found, and this pixel is marked as True to indicate that it has been visited; define an empty set To store the pixels of the current motion area, define the label as 1, start from the current pixel, query in the four directions of up, down, left and right. If the value of the adjacent pixel is also 1 and has not been visited, continue to query the adjacent position until the values around the pixels above and below the head and tail are all 0; mark all the passed pixels as True and add them to In the collection, The set and label are stored in the set Ω, label = label + 1, and the above steps are repeated until all pixels are traversed; the motion area set SP = (SP1, SP2, ..., SP N );

[0177] Step S2 specifically includes the following steps:

[0178] In this embodiment, the target detection is performed by using the image obtained by data acquisition. Figure 3 As shown, the gradient calculation template example is as follows Figure 4 As shown;

[0179] S201, performing noise reduction processing on the divided motion areas;

[0180] Use the filter window to filter the noise points in the image data. First, filter the suspected noise points in the image, calculate the grayscale average of each point in the window, and set the threshold T1 to filter the suspected noise. The formula of T1 is defined as follows:

[0181]

[0182] Among them, m, n represent the length and width of the filter window, h(i,j), Represents the grayscale value of pixel (i, j) and the mean grayscale value of the four area windows above, below, left and right respectively; when the condition h is not met max -T1 <h(i,j)<h max and h min <h(i,j)<h min At +T1, the pixel is a suspected noise point;

[0183] After filtering out the suspected noise points, When , this point is a noise point, and the calculation formula of the threshold T2 is as follows:

[0184]

[0185] Where P represents the number of noise points in the current window;

[0186] After obtaining the noise point, the image is denoised according to the number of noise points around the noise point. The noise point is denoised and its grayscale value is replaced. The formula is as follows:

[0187]

[0188] Among them, h represents the gray value of the pixel, h m is the gray value of the non-noise point in the window.

[0189] S202, using an n×n operator to design eight direction detection operators with the strong edge point (i, j) as the center to calculate the gradient amplitude and obtain the target strong edge information;

[0190] This embodiment uses a 5×5 operator template to select a strong edge point (i, j) as the center to design eight detection operators in the directions of 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. Figure 5 As shown;

[0191] The calculation formula of the gradient amplitude is:

[0192]

[0193] The gradient direction is arctan(G 90 / G0), where the subscript 0 of G0 represents the 0 degree direction, G 90 The subscript 90 indicates the 90-degree direction, and the same applies to the others.

[0194] In order not to lose edge details with weak local gradients, the local maximum pixels in the edge gradient direction are kept unchanged, and other pixels are set to 0;

[0195]

[0196] Where δ represents the mediation factor that can modify the degree of edge refinement;

[0197] S204, using the maximum inter-class variance method Otsu algorithm to connect the edges and form a complete target boundary;

[0198] First, the grayscale value ranges of the background area and the motion area are defined as two categories, namely: B1 = [0, k], B2 = [k, b], and the grayscale value range of the entire image is [0, b]. Then, the proportion of the number of pixels with grayscale values of the two categories relative to the global number of pixels is calculated, and the calculated result is multiplied by the average grayscale value of each category to obtain the inter-class variance formula Δ(k). The maximum inter-class variance is calculated iteratively to obtain the optimal segmentation threshold k. The inter-class variance calculation formula is as follows:

[0199]

[0200] B2=1-B1

[0201] Δ(k)=B1×b1 2 +B2×b2 2

[0202] Where b1 and b2 represent the average grayscale of B1 and B2 classes respectively;

[0203] S205, constructing a video stream image target tracking method based on the complete target boundary and the divided motion area;

[0204] After obtaining the complete boundary and motion area information of the target, the target object r0 is selected and d background images r around the target are extracted from its adjacent areas. i As a regularization term, it is introduced into the ridge regression equation, the context information is superimposed on the target area, and the target area is modeled. The calculation is as follows:

[0205]

[0206] Where g() represents the objective function, γ is the ridge coefficient, σ is the learned correlation filter, R represents the feature matrix of the target and the extracted surrounding background image, Represents the relationship matrix between the target and the surrounding background image, I n represents the identity matrix representation, represents the square of L2 norm;

[0207] The target response image is constructed based on the learned correlation filter, and the maximum response position detection is used to determine the position of the target in the current image. The detection formula is:

[0208]

[0209] In the formula, τ represents the regularization parameter to prevent overfitting, g w represents the surrounding background search window, ⊙ represents the point-by-point product operation;

[0210] According to the target results obtained by detection, two levels of target features are divided, and the deep features of the target position are defined as l d , the shallow features of the target details are l s , construct a weighted combination of features;

[0211] L(r)=λ1×l d (r)+λ2×l s (r)

[0212] λ1+λ2=1

[0213] Where r represents the target object, λ1 and λ2 represent the weights of deep features and shallow features respectively, and L represents the score value at the target position. After selecting the target, other frame entities are selected for comparison with the target entity, and the score value L(r*) of the candidate target is calculated. The predicted target is selected based on the difference in the score between the candidate target and the target. The minimization loss function Loss is defined as:

[0214]

[0215] In the formula, κ is the rate of change of the distance between two entities, θ() is a function that penalizes excessively large weight values, and e () represents the exponential function with base e;

[0216] Construct the time series data of the target and store the time and location information of the target during the tracking process in the form of time series data;

[0217] The model is updated based on the fluctuation of the response graph to ensure the continuity of the tracking process. When the target is disturbed, the response graph will fluctuate violently; when the target is not disturbed, the peak of the response graph is clearer. The dynamic evaluation index APCE is used to determine whether the model needs to be updated.

[0218]

[0219] Where y max Indicates the peak value of response, y min Represents the minimum response value, (w,h) represents the position of the target, and mean() represents the mean function.

[0220] The S3 specifically includes the following steps:

[0221] In this embodiment, an automatic image target annotation model is constructed based on the time series data collected during the tracking process. Figure 5 As shown;

[0222] S301, extracting the target's features in time and space dimensions based on the tracking results;

[0223] First, based on the collected time series data, the position sequence of the surrounding entities and the target entity in the search area at time t is calculated. Encode and generate embedding vectors. Encode the time steps first. The formula is as follows:

[0224]

[0225] Among them, e represents the parity of the time step, v represents the dimension of the timestamp feature; the position sequence and the time step sequence are added together to obtain the embedded sequence in time t

[0226] First, based on the collected time series data, the position sequence of the surrounding entities and the target entity in the search area at time t is calculated. Encode, generate embedding vectors, and encode the time steps. The formula is as follows:

[0227]

[0228] Among them, e represents the parity of the time step, v represents the dimension of the timestamp feature; the position sequence W is added to the time step sequence Encode(t) to obtain the embedded sequence within time t

[0229] In addition, by increasing the association relationship between the target entity and the surrounding entities at different time steps, context information and scene semantics can be obtained; for each time step t, the trajectory points of different entities are set to W t Input to the multi-head attention network, each multi-head attention network contains two multi-head attention layers, each multi-head attention layer uses G L Each subgraph attention layer focuses on the association relationship of the target entity in different subspaces. The association feature formula Z of the entity at each time step t is t as follows:

[0230]

[0231] Among them, μ is the shared parameter of the entity node, Entity represents the target entity, LeakyReLU() represents the activation function with leakage parameters, and [·||·] represents the concat connection operation of the result after the node is transformed. Indicates mapping the concatenated high-dimensional features into a real number. Indicates G L The related features of the sub-graph attention layers are fused;

[0232] Combine the sequence correlation features of all time steps into Perform embedding vector encoding on Z to generate spatial sequence Z w ;

[0233] S302: construct an automatic target labeling model based on the target characteristics, and construct a loss function of the model based on the automatic target labeling model and the training data;

[0234] The time series Y of the target entity is linearly divided into three vectors Q, K, and V, which are input into the multi-head self-attention layer to obtain the correlation of the target at different time steps. The output results are residually added and passed through the corresponding normalization layer to prevent the gradient from disappearing. The two layers of fully connected layers FCL are connected to output the kinematic feature Y of the target object. g ;

[0235] Attention1=Norm(Y+MultiHead-SelfAtt1(Q1,K1,V1;ψ1))

[0236]

[0237]

[0238] Y g =Norm(Attention1+FCL 2,out )

[0239] In the formula, ψ is the relevant parameter, ReLU() is the activation function, and Norm() represents the normalization function;

[0240] The associated feature Z of the target entity at all time steps w Perform linear differentiation to generate three matrices Q1, K1, and V1. After the following operations, the entity interaction feature Z is obtained g ;

[0241] Z g =Norm(Attention2+FCL 2,out )

[0242] Attention2=Norm(MultiHead-SelfAtt2(Q1,K1,V1;ψ2)+Z w )

[0243]

[0244]

[0245] Reasonably assign weights to motion features and interaction features to generate a comprehensive feature vector η, which is calculated as follows:

[0246] η=β1⊙Y g +β2⊙Z g

[0247]

[0248] β2=1-β1

[0249] in, Represents motion feature Y g The corresponding weight parameters, represents the corresponding bias parameter, ⊙ represents the point-by-point product operation;

[0250] The vector η is input into the fully connected layer for target classification and regression operations to determine the category of the target; the loss function calculation formula Loss1 is as follows:

[0251]

[0252] Among them, p i represents the true category of the target, p jRepresents the predicted category probability; sets the minimum bounding rectangle of the annotation box according to the position and size of the target, sets the annotation and color of the annotation box according to the category of the target, uses the same annotation box color for the same category, and stores the annotation information in the annotation dataset; ln() represents the logarithmic function; N represents the number of target objects.

[0253] S303. Use the loss function to calculate the gradient of each parameter in the model, and perform backpropagation to update the parameters in the model.

[0254] The S4 specifically includes the following steps:

[0255] In this embodiment, the example diagram of the annotation information obtained by the automatic annotation model of the embodiment is as follows: Figure 6 shown.

[0256] S401, extracting annotation information and generating a unique identifier;

[0257] The extracted annotation information is as follows Figure 6 As shown, Message1 = {timestamp, target set}, with timestamp as the primary key. The target set contains the total set of targets within this time, including target ID, target category, target location, target attribute, and target feature. The targets contained in all images within the same timestamp are consistent. The identifier {Num1, Num2, Num3, Num4, Num5, Num6, Num7, Num8, Num9, Num1} is generated according to the timestamp. i};

[0258] S402: Associating the annotation information with the unique identifier with the image data to ensure that each image has corresponding annotation information and establish a data correspondence relationship;

[0259] Among them, the image features are added to the annotation information, and a mapping function Φ1 is established to generate a new annotation information dataset Θ;

[0260] Ф1(message i )=[c1 1 (message i ),c2 1 (message i ),…,c u 1 (message i ),

[0261] c1 2 (message i ),c2 2 (message i ),…,c u 2(message i ),

[0262] …

[0263] c1 l (message i ),c2 l (message i ),…,c u l (message i )]

[0264] Among them, message i Represents the labeled information, i∈(1,u) represents the number of labeled information, and l represents the dimension;

[0265] S403: Integrate all images and corresponding annotation information according to the established data correspondence to generate structured data with correlation;

[0266] Among them, according to the correlation relationship in the labeled data, the correlation relationship matrix V, v ij is an element of V;

[0267]

[0268] Where, Respectively represent c i and c j The average value of and Respectively represent c i and c j The standard deviation of ; establish a mapping function Ф2 to fuse V with the new annotation information dataset Θ to generate a structured dataset with an associated relationship, such as Figure 7 shown.

[0269] The S5 specifically includes the following steps:

[0270] In this embodiment, the flowchart of using index to query structured data is as follows Figure 8 As shown;

[0271] S501, storing the structured data in a database according to the association relationship;

[0272] S502, extracting the key value of the key-value data into a data element form including the object word key.obj, the qualifier key.lim, and the property word key.par;

[0273] Among them, data element Key = {key.obj}∪{key.lim}∪{key.par};

[0274] S503. When data elements have the same key value but different key values, add the data item KE with the highest discrimination among the key values after the data element at that time; when the index positions are the same, store the corresponding data elements in the form of a linked list; and add a fine-grained locking mechanism in each linked list;

[0275] Among them, the data items are matched for similarity, and the comparison algorithm for obtaining the data item with the greatest discrimination in the key value is:

[0276]

[0277] Where Price() represents the cost parameter of converting two data items into the same data item, and length() represents the length of the data item;

[0278] S504. When querying data, first determine whether the data element can be queried within the specified number of times; if the data element exists, find the key value corresponding to the data element; finally, verify whether the key value meets the requirements. If the key value is consistent, return the key value corresponding to the key value.

[0279] In this embodiment, the video stream image data is fully extracted. Figure 2 and Figure 3 Not only does it extract target information, but it also combines the contextual information of the image to make the extracted data more accurate and comprehensive. Figure 6 The annotation information in the image uses the image target automatic annotation model to extract features from time and space, which can improve the analysis and understanding of the target, help add more dimensions of information to the collected structured data, contain richer data information, and facilitate data application and sharing.

[0280] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0281] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0282] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0283] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0284] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A structured acquisition method for edge video stream image data, characterized by: The method comprises the following steps: S1. Collect video stream image data and input it into the edge computing platform, segment and preprocess the video stream to obtain the target's motion range; S2. Build a target tracking method based on the target's motion range to obtain the target's time series data; S3. Build a target automatic labeling model based on the target's time series data, train the automatic labeling model, and update the corresponding model parameters; S4, associating the annotation information generated by the automatic annotation model after the updated parameters with the image, establishing a corresponding relationship between the data, and generating structured data with an associated relationship; S5. Store structured data in the database and add adaptive indexes for query optimization.

2. The structured acquisition method for edge video stream image data according to claim 1 is characterized in that : The step S1 specifically includes: S101, collecting video image data and inputting it into the edge computing platform; S102, storing the collected data in the form of a video sequence; S103, intercepting t frames of image from the continuous image frames each time for processing to reduce the amount of calculation and storage requirements; S104, inputting the captured images in sequence and performing grayscale processing on the images; S105: Divide the target's motion range by extracting a background template.

3. The structured acquisition method for edge video stream image data according to claim 1, characterized in that: The step S105 specifically includes: The first input frame X1 is selected as the reference frame, and the vector x1(i, j) represents the grayscale value of the pixel (i, j) of the first frame image, where i = 1, 2, 3, ..., M, j = 1, 2, 3, ..., N, M represents the width of the input video frame, N represents the height of the input video frame, and ... t (i, j) represents the grayscale value of the pixel (i, j) in the t-th frame image, t is a positive integer greater than or equal to 2, and t consecutive video frames are represented by the vector set [x1, x2, x3, ..., x t ] is used to represent the gray value difference f between the t-th frame image and the reference frame at pixel (i, j) 1t The calculation formula is as follows: f 1t＝ ||x t (i,j)|-|x1(i,j)|| The above formula is used to compare the grayscale value difference of the continuous t-frame image with the reference frame at the pixel (i, j). The largest value F is selected from them to determine whether the grayscale value of the pixel has changed, thereby determining the area of this pixel. The F value calculation formula is as follows: F=max{f 12 ,f 13 ,…,f 1t } According to the above formula, the maximum grayscale difference value of all pixels is stored in the matrix S M×N , when F is 0, it indicates that the pixel has not changed and is stored in S with a value of 0; when F is not equal to 0, let and are the maximum and minimum grayscale values of the pixel in the t-th frame compared with the previous t-1 frame; set the threshold T, when and When the difference value corresponding to the pixel point is less than the threshold, it is classified as the background area. i,j Set to 0; Among them, S i,j Represents the binary image at pixel (i, j), if S i,j If the value of is 1, then this pixel does not belong to the background area, and the pixels with a value of 1 are connected to generate a motion area.

4. The structured acquisition method for edge video stream image data according to claim 1, characterized in that: The step S2 specifically includes: S201, performing noise reduction processing on the divided motion areas; S202, setting a filter window of size n×n, taking the strong edge point (i, j) as the center, using detection operators in eight directions to calculate the gradient amplitude, and obtaining the target strong edge information; The calculation formula of the gradient amplitude is: The gradient direction is arctan(G 90 / G0), where the subscript 0 of G0 represents the 0 degree direction, G 90 The subscript 90 indicates the 90-degree direction, and the same applies to the others. S203, retaining edge details of the local gradient to obtain the edge structure of the target to be identified in the image; Among them, the adjacent pixels in the gradient direction of each pixel are checked, and only the local maximum values of the edge are retained; Where δ represents the mediation factor that can modify the degree of edge refinement; S204, using the maximum inter-class variance method to connect the edges and form a complete target boundary; S205: Construct a video stream image target tracking method based on the complete target boundary and the divided motion area.

5. The structured acquisition method for edge video stream image data according to claim 4, characterized in that: The noise reduction processing method in step S201 specifically includes: Use the filter window to filter the noise points in the image data. First, filter the suspected noise points in the image, calculate the grayscale average of each point in the window, and set the threshold T1 to filter the suspected noise. The formula of T1 is defined as follows: Where m and n represent the length and width of the filter window, respectively, h(i,j), Represents the grayscale value of pixel (i, j) and the mean grayscale value of the four area windows above, below, left and right respectively; when the condition h is not met max -T1 <h(i,j)<h max and h min <h(i,j)<h min At +T1, the pixel is a suspected noise point; After filtering out the suspected noise points, When , this point is a noise point, and the calculation formula of the threshold T2 is as follows: Where P represents the number of noise points in the current window; After obtaining the noise point, the image is denoised according to the number of noise points around the noise point. The noise point is denoised and its grayscale value is replaced. The formula is as follows: Among them, h represents the gray value of the pixel, h m is the gray value of the non-noise point in the window.

6. The structured acquisition method for edge video stream image data according to claim 4, characterized in that: The target tracking method in step S205 specifically includes: After obtaining the complete boundary and motion area information of the target, the target object r0 is selected and d background images r around the target are extracted from its adjacent areas. i As a regularization term, it is introduced into the ridge regression equation, the context information is superimposed on the target area, and the target area is modeled. The calculation is as follows: Where g() represents the objective function, γ is the ridge coefficient, σ is the learned correlation filter, R represents the feature matrix of the target and the extracted surrounding background image, Represents the relationship matrix between the target and the surrounding background image, I n represents the identity matrix representation, represents the square of L2 norm; The target response image is constructed based on the learned correlation filter, and the maximum response position detection is used to determine the position of the target in the current image. The detection formula is: In the formula, τ represents the regularization parameter to prevent overfitting, g w represents the surrounding background search window, ⊙ represents the point-by-point product operation; According to the target results obtained by detection, two levels of target features are divided, and the deep features of the target position are defined as l d , the shallow features of the target details are l s , construct a weighted combination of features; L(r)=λ1×l d (r)+λ2×l s (r) λ1+λ2=1 Where r represents the target object, λ1 and λ2 represent the weights of deep features and shallow features respectively, and L represents the score value at the target position. After selecting the target, other frame entities are selected for comparison with the target entity, and the score value L(r*) of the candidate target is calculated. The predicted target is selected based on the difference in the score between the candidate target and the target. The minimization loss function Loss is defined as: In the formula, κ is the rate of change of the distance between two entities, θ() is a function that penalizes excessively large weight values, and e () represents the exponential function with base e; Construct the time series data of the target and store the time and location information of the target during the tracking process in the form of time series data; The model is updated based on the fluctuation of the response graph to ensure the continuity of the tracking process. When the target is disturbed, the response graph will fluctuate violently; when the target is not disturbed, the peak of the response graph is clearer. The dynamic evaluation index APCE is used to determine whether the model needs to be updated. Where y max Indicates the peak value of response, y min Represents the minimum response value, (w,h) represents the position of the target, and mean() represents the mean function.

7. The structured acquisition method and system for edge video stream image data according to claim 1, characterized in that: The step S3 specifically includes: S301, extracting the target's features in time and space dimensions based on the tracking results; First, based on the collected time series data, the position sequence of the surrounding entities and the target entity in the search area at time t is calculated. Encode, generate embedding vectors, and encode the time steps. The formula is as follows: Among them, e represents the parity of the time step, v represents the dimension of the timestamp feature; the position sequence W is added to the time step sequence Encode(t) to obtain the embedded sequence within time t In addition, by increasing the association relationship between the target entity and the surrounding entities at different time steps, context information and scene semantics can be obtained; for each time step t, the trajectory points of different entities are set to W t ={W1 t ,W2 t ,…,W g t }, W t Input to the multi-head attention network, each multi-head attention network contains two multi-head attention layers, each multi-head attention layer uses G L Each subgraph attention layer focuses on the association relationship of the target entity in different subspaces. The association feature formula Z of the entity at each time step t is t as follows: Among them, μ is the shared parameter of the entity node, Entity represents the target entity, LeakyReLU() represents the activation function with leakage parameters, [·||·] represents the concat connection operation of the result after the node transformation, and a represents mapping the concatenated high-dimensional features into a real number. Indicates G L The related features of the sub-graph attention layer are fused; Combine the sequence correlation features of all time steps into Perform embedding vector encoding on Z to generate spatial sequence Z w ; S302: construct an automatic target labeling model based on the target characteristics, and construct a loss function of the model based on the automatic target labeling model and the training data; S303: Use the loss function to calculate the gradient of each parameter in the model, and perform backpropagation to update the parameters in the model.

8. The structured acquisition method and system for edge video stream image data according to claim 7, characterized in that: The step S302 constructs an automatic target labeling algorithm model, specifically including: The time series Y of the target entity is linearly divided into three vectors Q, K, and V, which are input into the multi-head self-attention layer to obtain the correlation of the target at different time steps. The output results are residually added and passed through the corresponding normalization layer to prevent the gradient from disappearing. The two layers of fully connected layers FCL are connected to output the kinematic feature Y of the target object. g ; Attention1=Norm(Y+MultiHead-SelfAtt1(Q1,K1,V1;ψ1)) FCL 1,out =ReLU(FCL1(Attention2;ψ FCL1 )) FCL 2,out =FCL2(FCL 1,out ;ψ FCL2 ) Y g =Norm(Attention1+FCL 2,out ) In the formula, ψ is the relevant parameter, ReLU() is the activation function, and Norm() represents the normalization function; The associated feature Z of the target entity at all time steps w Perform linear differentiation to generate three matrices Q1, K1, and V1. After the following operations, the entity interaction feature Z is obtained g ; Z g =Norm(Attention2+FCL 2,out ) Attention2=Norm(MultiHead-SelfAtt2(Q1,K1,V1;ψ2)+Z w ) FCL 2,out =FCL2(FCL 1,out ;ψ FCL2 ) FCL 1,out =ReLU(FCL1(Attention2;ψ FCL1 )) Reasonably assign weights to motion features and interaction features to generate a comprehensive feature vector η, which is calculated as follows: η=β1⊙Y g +β2⊙Z g β1=sigmoid(ζ Yg +c Yg ) β2＝1-β1 Among them, Yg Represents motion feature Y g The corresponding weight parameter, c Yg represents the corresponding bias parameter, ⊙ represents the point-by-point product operation; The vector η is input into the fully connected layer for target classification and regression operations to determine the category of the target; the loss function calculation formula Loss1 is as follows: Among them, p i represents the true category of the target, p j Represents the predicted category probability; sets the minimum bounding rectangle of the annotation box according to the position and size of the target, sets the annotation and color of the annotation box according to the category of the target, uses the same annotation box color for the same category, and stores the annotation information in the annotation dataset; ln() represents the logarithmic function; N represents the number of target objects.

9. The structured acquisition method and system for edge video stream image data according to claim 1, characterized in that: The step S4 specifically includes: S401, extracting annotation information and generating a unique identifier; S402: Associating the annotation information with the unique identifier with the image data to ensure that each image has corresponding annotation information and establish a data correspondence relationship; Among them, the image features are added to the annotation information, and a mapping function Φ1 is established to generate a new annotation information dataset Θ; Ф1(message i )=[c1 1 (message i ),c2 1 (message i ),…,c u 1 (message i ), c1 2 (message i ),c2 2 (message i ),…,c u 2 (message i ), … c1 l (message i ),c2 l (message i ),…,c u l (message i )] Among them, message i Represents the labeled information, i∈(1,u) represents the number of labeled information, and l represents the dimension; S403: Integrate all images and corresponding annotation information according to the established data correspondence to generate structured data with correlation; Among them, according to the correlation relationship in the labeled data, the correlation relationship matrix V, v ij is an element of V; Where, Respectively represent c i and c j The average value of and Respectively represent c i and c j The standard deviation of V is established; a mapping function Ф2 is established to fuse V with the new annotation information dataset Θ to generate a structured dataset with an associated relationship.

10. The structured acquisition method for edge video stream image data according to claim 1, characterized in that: The step S5 specifically includes: S501, storing the structured data in a database according to the association relationship; S502, extracting the key value of the key-value data into a data element form including the object word key.obj, the qualifier key.lim, and the property word key.par; Among them, data element Key = {key.obj}∪{key.lim}∪{key.par}; S503. When data elements have the same key value but different key values, add the data item KE with the highest discrimination among the key values after the data element at that time; when the index positions are the same, store the corresponding data elements in the form of a linked list; and add a fine-grained locking mechanism in each linked list; Among them, the data items are matched for similarity, and the comparison algorithm for obtaining the data item with the greatest discrimination in the key value is: Where Price() represents the cost parameter of converting two data items into the same data item, and length() represents the length of the data item. S504. When querying data, first determine whether the data element can be queried within the specified number of times; if the data element exists, find the key value corresponding to the data element; finally, verify whether the key value meets the requirements. If the key value is consistent, return the key value corresponding to the key value.

Images