Cloud-edge collaborative intelligent video processing method

Abstract

The invention discloses a cloud-edge collaborative intelligent video processing method, and the method comprises the following steps: S1, carrying out the preliminary analysis and processing of a real-time video stream through edge end equipment, and extracting necessary image information; s2, the district and county-level platform analyzes the necessary picture information by using a local algorithm, extracts feature vectors, locally stores the feature vectors, and sends the feature vectors to the provincial-level cloud platform; s3, the provincial level cloud platform receives the feature vectors sent by the district and county level platforms, performs management and control scheduling on each district and county level platform, processes data of the district and county level platform with data overstock and the highest task priority, and schedules the data to the computing node with the lowest load; and S4, the provincial cloud platform uses the vector data in the vector storage center to analyze and / or provide search service.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, and specifically relates to a cloud-edge collaborative intelligent video processing method. Background Art

[0002] With the surge in the number of cameras, a large amount of video data is generated, which poses a challenge to data transmission and central processing capabilities. The traditional single-center data processing model is difficult to efficiently cope with these challenges, so a distributed, cloud-edge collaborative video processing method is needed to adapt to the growing demand and ensure the real-time and reliability of video analysis. Existing solutions usually focus on the advantages of local edge computing + cloud computing. Edge computing solutions can reduce bandwidth usage, but the computing power of each region seems to be insufficient when processing large-scale data. If a unified cloud computing solution is established with powerful processing capabilities, the transmission of large amounts of data will put tremendous pressure on network bandwidth and lack real-time performance. Summary of the invention

[0003] The present invention provides a cloud-edge collaborative intelligent video processing method, the method comprising the following steps:

[0004] S1. The edge device performs preliminary analysis and processing on the real-time video stream and extracts necessary image information;

[0005] S2. The district and county-level platform uses local algorithms to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform;

[0006] S3. The provincial cloud platform receives the feature vector sent by the district and county platforms and stores it in the vector storage center. The control center of the provincial cloud platform controls and dispatches each district and county platform. If a backlog occurs on the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling and processing. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes, and the load status of each computing node, the data backlog of each district and county platform, and the task priority of each district and county platform are obtained. The data processing of the district and county platform with data backlog and the highest task priority is dispatched to the computing node with the lowest load. The data processing includes the data processing process of parsing the necessary image information and extracting the feature vector in step S2;

[0007] S4. The provincial cloud platform uses the vector data in the vector storage center to perform analysis and / or provide search services.

[0008] Further, step S1 specifically includes: acquiring each frame image in the real-time video stream, and taking the frame image whose difference with the adjacent frame exceeds a threshold as the necessary frame image, and extracting the information of the necessary frame image as the necessary picture information.

[0009] Further, step S2 specifically includes: extracting key point descriptor vectors from the image information using a local feature point extraction algorithm, calculating the residual sum of the key points belonging to each cluster center and the cluster center to which it belongs, and obtaining a feature vector based on the residual sum.

[0010] Further, step S1 includes the following calculation process:

[0011]

[0012] in, is the tth frame image in the real-time video stream, is the t-th frame image in the image The pixel value at the pixel location, and are the width and height of the image, is the threshold parameter set, Necessary picture information.

[0013] Further, step S2 includes the following calculation process:

[0014]

[0015]

[0016] in, is the feature vector of necessary image information, which is The splicing, is the vector corresponding to the cluster center, To use SIFT local feature point extraction algorithm to extract necessary image information The extracted cluster center Each key point descriptor vector of The cluster center The number of key point descriptor vectors, is the number of cluster centers, is the L2 normalization function, Indicates that all the cluster centers The key point descriptor vector and cluster center The residual sum.

[0017] Furthermore, the control and scheduling includes calculating the control and scheduling result R,

[0018]

[0019] in, Indicates The queue status of each district and county-level platform is the data backlog. Indicates Task priority parameters for each district and county-level platform, Indicates the number of The load of each computing node is measured, and D is the scheduling function, which is used to schedule the district and county-level platform tasks with data backlog and high task priority to the node with the lowest load in the dynamic computing power pool.

[0020] Furthermore, providing the search service specifically includes the following processing steps:

[0021]

[0022]

[0023] in, is the query vector, is the feature vector of the image, is the vector dimension, The query vector The value of each dimension, is the image feature vector The value of each dimension of is the similarity result, Return results for the search. It is a sorting function that sorts the results according to similarity and returns search results with high similarity.

[0024] The present invention also relates to a system for implementing the above-mentioned cloud-edge collaborative intelligent video processing method, the system comprising: a provincial cloud platform, multiple district and county platforms, and multiple edge devices;

[0025] The edge device is connected to at least one camera for collecting real-time video streams, and performs preliminary analysis and processing on the real-time video streams to extract necessary image information;

[0026] The district and county-level platform is connected to at least one of the edge segment devices, and uses a local algorithm to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform;

[0027] The provincial cloud platform includes a control center, a vector storage center, a dynamic computing power pool and service applications. The provincial cloud platform receives the feature vectors sent by the district and county platforms and stores them in the vector storage center. The control center manages and dispatches each district and county platform. If a backlog occurs on the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes, and the load status of each computing node, the data backlog of each district and county platform and the task priority of each district and county platform are obtained. The data processing of the district and county platform with data backlog and the highest task priority is dispatched to the computing node with the lowest load. The data processing includes the district and county platform parsing the necessary image information, extracting the feature vector, and performing the data processing process for storage;

[0028] The service application uses the vector data in the vector storage center to perform analysis and / or provide search services.

[0029] The present invention also relates to a computer program product, which includes a computer program, and the computer program is executed by a processor to execute the cloud-edge collaborative intelligent video processing method as described above.

[0030] The present invention also relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor to execute the cloud-edge collaborative intelligent video processing method as described above.

[0031] The technical solution of the present invention sets up a three-level distributed processing architecture including edge terminals, district and county-level platforms, and provincial cloud platforms. Through the scheduling processing of the dynamic computing power pool of the provincial cloud platform, it not only retains the low latency and real-time performance of edge computing and regional clouds, but also takes into account the efficient analysis capabilities of central-level cloud computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the system structure of an intelligent video processing method for realizing cloud-edge collaboration according to the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the present invention.

[0034] Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those of ordinary skill in the art to which the present invention belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0035] Embodiment 1 of the present invention relates to a cloud-edge collaborative intelligent video processing method, the method comprising the following steps:

[0036] S1. The edge device performs preliminary analysis and processing on the real-time video stream to extract necessary image information. Specifically, it obtains each frame image in the real-time video stream, and takes the frame image whose difference with the adjacent frame exceeds the threshold as the necessary frame image, and extracts the information of the necessary frame image as the necessary image information. The specific processing method is:

[0037]

[0038] in, is the tth frame image in the real-time video stream, is the t-th frame image in the image The pixel value at the pixel location, and are the width and height of the image, is the threshold parameter set, Necessary picture information.

[0039] S2. The district and county-level platform uses local algorithms to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform. Specifically, it uses a local feature point extraction algorithm to extract the key point descriptor vectors in the image information, calculates the residual sum of the key points belonging to each cluster center and the cluster center to which it belongs, and obtains the feature vector based on the residual sum. The specific calculation process is as follows:

[0040]

[0041]

[0042] in, is the feature vector of necessary image information, which is The splicing, is the vector corresponding to the cluster center, To use SIFT local feature point extraction algorithm to extract necessary image information The extracted cluster center Each key point descriptor vector of The cluster center The number of key point descriptor vectors, is the number of cluster centers, is the L2 normalization function, Indicates that all the cluster centers The key point descriptor vector and cluster center The residual sum.

[0043] S3. The provincial cloud platform receives the feature vectors sent by the district and county platforms and stores them in the vector storage center. The management and control center of the provincial cloud platform manages and dispatches each district and county platform. If a backlog occurs at the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling and processing. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes. The load status of each computing node, the data backlog of each district and county platform, and the task priority of each district and county platform are obtained. The data processing of the district and county platform with data backlog and the highest task priority is scheduled to the computing node with the lowest load. The data processing includes the data processing process of parsing the necessary image information and extracting the feature vector in step S2.

[0044] The control and scheduling includes calculating the control and scheduling result R,

[0045]

[0046] in, Indicates The queue status of each district and county-level platform is the data backlog. Indicates Task priority parameters for each district and county-level platform, Indicates the number of The load of each computing node is measured, and D is the scheduling function, which is used to schedule the district and county-level platform tasks with data backlog and high task priority to the node with the lowest load in the dynamic computing power pool.

[0047] S4. The provincial cloud platform uses the vector data in the vector storage center to perform analysis and / or provide search services.

[0048] Taking the provision of search services as an example, the specific processing includes the following:

[0049]

[0050]

[0051] in, is the query vector, is the feature vector of the image, is the vector dimension, The query vector The value of each dimension, is the image feature vector The value of each dimension of is the similarity result, Return results for the search. It is a sorting function that sorts the results according to similarity and returns search results with high similarity.

[0052] Embodiment 1 of the present invention relates to a system for implementing a cloud-edge collaborative intelligent video processing method of embodiment 1, as shown in the attached Figure 1 As shown, the system includes a provincial cloud platform, multiple district and county platforms, and multiple edge devices. The edge device is connected to at least one camera that collects real-time video streams, and performs preliminary analysis and processing on the real-time video streams to extract necessary image information. The district and county platform is connected to at least one of the edge devices, and uses a local algorithm to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform. The provincial cloud platform includes a control center, a vector storage center, a dynamic computing power pool, and a service application. The provincial cloud platform receives the feature vectors sent by the district and county platforms and stores them in the vector storage center. The control center manages and dispatches each district and county platform. If a backlog occurs on the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes, which obtain the load status of each computing node, the data backlog of each district and county platform, and the task priority of each district and county platform. The data processing of the district and county platform with data backlog and the highest task priority is dispatched to the computing node with the lowest load. The data processing includes the district and county platform parsing the necessary image information, extracting the feature vector, and storing the data processing process. The service application uses the vector data in the vector storage center to analyze and / or provide search services.

[0053] Embodiment 3 of the present invention relates to a computer program product, which includes a computer program. The computer program is executed by a processor and is used to execute a cloud-edge collaborative intelligent video processing method of Embodiment 1.

[0054] Embodiment 4 of the present invention relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor to execute a cloud-edge collaborative intelligent video processing method of Embodiment 1.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A cloud-edge collaborative intelligent video processing method, characterized in that: The method comprises the following steps: S1. The edge device performs preliminary analysis and processing on the real-time video stream and extracts necessary image information; S2. The district and county-level platform uses local algorithms to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform; S3. The provincial cloud platform receives the feature vector sent by the district and county platforms and stores it in the vector storage center. The control center of the provincial cloud platform controls and dispatches each district and county platform. If a backlog occurs on the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling and processing. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes, and the load status of each computing node, the data backlog of each district and county platform, and the task priority of each district and county platform are obtained. The data processing of the district and county platform with data backlog and the highest task priority is dispatched to the computing node with the lowest load. The data processing includes the data processing process of parsing the necessary image information and extracting the feature vector in step S2; S4. The provincial cloud platform uses the vector data in the vector storage center to perform analysis and / or provide search services.

2. The cloud-edge collaborative intelligent video processing method according to claim 1, characterized in that: Step S1 specifically includes: acquiring each frame image in the real-time video stream, and taking the frame image whose difference with the adjacent frame exceeds a threshold as the necessary frame image, and extracting the information of the necessary frame image as the necessary picture information.

3. The cloud-edge collaborative intelligent video processing method according to claim 1, characterized in that: Step S2 specifically includes: extracting key point descriptor vectors from the image information using a local feature point extraction algorithm, calculating the residual sum between the key points belonging to each cluster center and the cluster center to which it belongs, and obtaining a feature vector based on the residual sum.

4. The cloud-edge collaborative intelligent video processing method according to claim 2, characterized in that: Step S1 includes the following calculation process: ； in, is the t-th frame image in the real-time video stream, is the t-th frame image in the image The pixel value at the pixel location, and are the width and height of the image, is the threshold parameter set, Necessary picture information.

5. The cloud-edge collaborative intelligent video processing method according to claim 3, characterized in that: Step S2 includes the following calculation process: ； ； in, is the feature vector of necessary image information, which is The splicing, is the vector corresponding to the cluster center, To use SIFT local feature point extraction algorithm to extract necessary image information The extracted data belongs to the cluster center Each key point descriptor vector of The cluster center The number of key point descriptor vectors, is the number of cluster centers, is the L2 normalization function, Indicates that all the cluster centers The key point descriptor vector and cluster center The residual sum.

6. A cloud-edge collaborative intelligent video processing method according to any one of claims 1 to 5, characterized in that: The control and scheduling includes calculating the control and scheduling result R, ； in, Indicates The queue status of each district and county-level platform is the data backlog. Indicates Task priority parameters for each district and county-level platform, Indicates the number of The load of each computing node is measured, and D is the scheduling function, which is used to schedule the district and county-level platform tasks with data backlog and high task priority to the node with the lowest load in the dynamic computing power pool.

7. A cloud-edge collaborative intelligent video processing method according to any one of claims 1 to 5, characterized in that: Providing search services specifically includes the following processing: ； ； in, is the query vector, is the feature vector of the image, is the vector dimension, The query vector The value of each dimension, is the image feature vector The value of each dimension of is the similarity result, Return results for the search. It is a sorting function that sorts the results according to similarity and returns search results with high similarity.

8. A system for implementing a cloud-edge collaborative intelligent video processing method as described in any one of claims 1 to 7, characterized in that: The system includes: a provincial cloud platform, multiple district and county platforms, and multiple edge devices; The edge device is connected to at least one camera for collecting real-time video streams, and performs preliminary analysis and processing on the real-time video streams to extract necessary image information; The district and county-level platform is connected to at least one of the edge segment devices, and uses a local algorithm to parse the necessary image information, extract feature vectors, store them locally, and send them to the provincial cloud platform; The provincial cloud platform includes a control center, a vector storage center, a dynamic computing power pool and service applications. The provincial cloud platform receives the feature vectors sent by the district and county platforms and stores them in the vector storage center. The control center manages and dispatches each district and county platform. If a backlog occurs on the district and county platform, the dynamic computing power pool of the provincial cloud platform is used for scheduling. The dynamic computing power pool of the cloud computing platform of the provincial cloud platform includes multiple computing nodes, and the load status of each computing node, the data backlog of each district and county platform and the task priority of each district and county platform are obtained. The data processing of the district and county platform with data backlog and the highest task priority is dispatched to the computing node with the lowest load. The data processing includes the district and county platform parsing the necessary image information, extracting the feature vector, and performing the data processing process for storage; The service application uses the vector data in the vector storage center to perform analysis and / or provide search services.

9. A computer program product, characterized in that The computer program product includes a computer program, which is executed by a processor and is used to execute a cloud-edge collaborative intelligent video processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is executed by a processor to execute a cloud-edge collaborative intelligent video processing method as described in any one of claims 1-7.

Images