End-to-end cloud collaborative method and system based on visual scene perception

Abstract

The present invention provides an end-edge-cloud collaboration method based on visual scene perception, wherein the end-edge-cloud is an intelligent terminal, an edge server and a cloud server, comprising the following steps: the intelligent terminal perceives the complexity of the visual scene; the video task to be processed is divided into a simple video task, a medium video task and a complex video task according to the complexity of the visual scene; for the simple video task, the intelligent terminal transmits the video task to the edge server for calculation; for the complex video task, the intelligent terminal transmits the video task to the cloud server for calculation; and for the medium video task, the intelligent terminal distributes the video task between the edge server and the cloud server according to the ratio of available resources of the edge server and the cloud server.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of Internet of Things, and relates to a visual perception cooperation method in Internet of Things, in particular to an end-edge-cloud cooperation processing method and system for large-scale visual Internet of Things devices. BACKGROUND

[0002] In recent years, with the rapid development of technologies such as Internet of Things, edge computing and artificial intelligence, everything connected has become an inevitable trend that can be foreseen. Internet of Things terminals have massive data, and video is the largest type of data stored in terminals. Visual Internet of Things is the visual information perception part of Internet of Things, which uses various cameras to collect video data of people, vehicles and objects, and uses intelligent analysis technology to process visual information to provide support for subsequent analysis. However, when facing resource-intensive video tasks, if all video data is processed on the cloud, even if the cloud server has very high computing power, due to limited bandwidth resources, a large amount of data may cause network congestion, making it difficult to meet the requirements of tasks with ultra-low latency. Under the driving of deep neural networks, edge intelligence has been widely applied in Internet of Things scenarios, mainly for inference services in various new scenarios in smart cities, including intelligent transportation, smart communities, industrial Internet of Things, etc. The increasing popularity of Internet of Things devices with visual information collection capabilities will lead to a large amount of data flowing from the cloud to the edge. Current research processes massive video data on the edge to upload key information and complex information data sets to the cloud, reducing the load of the cloud server, while edge computing also increases real-time performance. Edge computing has a great advantage in saving transmission bandwidth because it is more dispersed and closer to the data source. However, the computing resources of edge devices are limited, and they can usually only handle some simple inference services, making it difficult to meet the accuracy requirements of complex inference services.

[0003] Current research on end-edge-cloud cooperation is limited to basic technologies such as task migration and resource allocation. In the context of increasingly intelligent Internet of Things devices, it is difficult to meet the more efficient data cooperation processing requirements. For example, some research uses an intelligent task distributor to balance the load between different computing nodes, analyzes the generated task information from the algorithms on the edge and cloud sides, and feeds back to the camera, then improves the processing quality of the task under resource constraints. Some research introduces a catastrophic strategy to simulate the phenomenon of disaster in biological evolution, thereby reducing the task processing delay; some research optimizes the overall delay according to the waiting time of the task and the execution interval between tasks; some research uses a double-time-scale optimization algorithm to minimize the cost of task offloading while ensuring acceptable service delay. However, these studies only consider the resource consumption between servers, ignoring the complexity characteristics of specific scenarios and terminal data, which may result in poor real-time performance and resource waste due to low cooperation efficiency when facing large-scale video data. SUMMARY

[0004] Therefore, the purpose of the present application is to overcome the defects of the prior art described above, to provide an end-edge cloud collaborative method and system based on visual scene perception.

[0005] According to a first aspect of the present application, an end-edge cloud collaborative method based on visual scene perception is provided, the end-edge cloud being an intelligent terminal, an edge server and a cloud server, comprising the following steps:

[0006] S1: the intelligent terminal perceives the visual scene complexity,

[0007] S2: according to the visual scene complexity, the video task to be processed is divided into a simple video task, a medium video task and a complex video task;

[0008] S3: for the simple video task, the intelligent terminal transmits the video task to the edge server for calculation; for the complex video task, the intelligent terminal transmits the video task to the cloud server for calculation; and for the medium video task, the intelligent terminal allocates the video task between the edge server and the cloud server according to the available resource proportion of the edge server and the cloud server.

[0009] According to an embodiment of the present application, the scene complexity C is calculated according to the following formula:

[0010] C = a · C g + b · C l

[0011] Wherein, C g is the global complexity, C l is the local complexity, and a, b are the weights of the global complexity and the local complexity, respectively.

[0012] According to an embodiment of the present application, the division method of the simple video task, the medium video task and the complex video task is: when the scene complexity is lower than or equal to a preset threshold C1, it is a simple video task; when the scene complexity is higher than or equal to a preset threshold C2, it is a complex video task; when the scene complexity is higher than the preset threshold C1 and lower than the preset threshold C2, it is a medium video task, wherein C1 is less than C2.

[0013] According to an embodiment of the present application, the available resource proportion calculation method of the edge server is

[0014]

[0015] Wherein, I e , B e , G e , D erespectively, are the available proportions of computing power resources, bandwidth resources, memory resources and storage space of the edge server in the current state,

[0016] The available resource proportion calculation method of the cloud server is

[0017]

[0018] wherein, I c , B c , G c , D c respectively are the available proportions of computing power resources, bandwidth resources, memory resources and storage space of the cloud server in the current state.

[0019] According to an embodiment of the present application, the method for distributing video tasks is to compare the task coefficient Tc with the server available resource coefficient SRc,

[0020] If T C ≤ SR C , the video task is uploaded to the edge server for processing,

[0021] If T C > SR C , the video task is uploaded to the cloud server for processing,

[0022] wherein C1 and C2 are preset threshold values and C1 < C2,

[0023] According to an embodiment of the present application, the calculation method of the global complexity C g of the image is as follows:

[0024] C g = α1·H + α2·SF + α3·φ

[0025] wherein, H is information entropy, SF is spatial frequency, φ is noise standard deviation, and α1, α2 and α3 are weights of information entropy, spatial frequency and noise standard deviation respectively.

[0026] According to an embodiment of the present application, the calculation method of the information entropy H is as follows:

[0027]

[0028] wherein, P i,j is the occurrence probability of the gray value of the feature binary tuple in the image.

[0029] According to an embodiment of the present application, the calculation method of the spatial frequency SF is as follows:

[0030]

[0031]

[0032]

[0033] wherein RF is spatial row frequency, CF is spatial column frequency, m, n are the number of pixels in width direction and height direction of the image respectively, and y(i,j) is the gray value of the image in the ith row and jth column.

[0034] According to one embodiment of the present application, the local complexity C of the image is calculated as follows: l

[0035] C l = β1·ΔH + β2·Δθ

[0036] wherein ΔH is mutual information, Δθ is contrast difference, and β1, β2 are the weights of mutual information and contrast difference respectively.

[0037] According to one embodiment of the present application, the mutual information ΔH is calculated as follows:

[0038] ΔH = H t + H b - H t,b

[0039] wherein H t and H b are the information entropy of the target region and the background region respectively, and H t,b is joint entropy.

[0040] According to one embodiment of the present application, the contrast difference is calculated as follows:

[0041]

[0042] wherein μ t is the mean of gray value of the target region, μ b is the mean of gray value of the background region, and σ is the standard deviation of gray value of the target region.

[0043] According to a second aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement any of the above described methods.

[0044] According to a third aspect of the present application, there is provided a computing system comprising: a storage device, and one or more processors; wherein the storage device is configured to store one or more computer programs, the computer programs being executable by the processors to implement any of the above described methods.​

[0045] Compared with the prior art, the application has the advantages that the scene perception capability of the intelligent terminal and the server resource allocation are combined for comprehensive analysis, each video task is reasonably allocated to the cloud server and the edge server according to the analysis result, and then the collaborative processing of large-scale video data is realized and the operation efficiency of the whole system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] The embodiments of the application will be further described below with reference to the drawings, in which:

[0047] Figure 1 A flow chart of edge-cloud collaborative task processing according to the embodiments of the application;

[0048] Figure 2 A scene complexity perception block diagram according to the embodiments of the application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0050] The scene sample data of a large number of video monitoring is complex and changeable, mainly in different quantities, different distances, different angles, different light, etc., so it is difficult to reasonably allocate. How to allocate the edge-cloud collaboration according to the characteristics of different scenes is crucial to improve the processing efficiency of video data.

[0051] The inventor found that in practical application, some simple scene data samples with less effective information are also uploaded to the cloud server for processing, resulting in waste of cloud server resources; and more and more Internet of Things devices have certain intelligent processing capability. Therefore, the inventor proposes an edge-cloud collaborative task processing method, and its processing flow is as shown in Figure 1 , and

[0052] A light-weight model is deployed on the intelligent terminal to perceive the scene complexity, and then the data of simple tasks is uploaded to the edge server for processing according to the perception result, and the data of complex tasks is uploaded to the cloud server for processing, and the medium tasks are combined with resource comprehensive analysis, the resources include various resources of the edge server and the cloud server, mainly computing resources, bandwidth resources and cache resources. The edge server and the cloud server output the corresponding results after processing the data uploaded by the intelligent terminal.

[0053] This paper will elaborate the application from four aspects of visual scene complexity perception, server available resource analysis, video task collaborative allocation scheme and specific implementation steps.

[0054] I. Scene complexity perception

[0055] The intelligent terminal scene complexity perception block diagram is shown in Figure 2 The input data is processed from the global and local two angles to obtain the global complexity and local complexity, wherein the parameters perceived to determine the global complexity are information entropy, spatial frequency and noise standard deviation, and the parameters perceived to determine the local complexity are mutual information and contrast difference. The visual task scene complexity is determined according to the perceived global complexity and local complexity.

[0056] 1. Global complexity

[0057] The global complexity of the image is represented by information entropy, spatial frequency and noise standard deviation.

[0058] 1) Information entropy

[0059] The pixel point of the image and the neighborhood information of the point constitute a feature pair, denoted as (i, j), wherein i represents the gray value of the pixel, and j represents the neighborhood gray mean value. The occurrence probability of the gray value of the feature pair in the image is defined as:

[0060]

[0061] Wherein f(i, j) is the number of times of the feature pair, and W and H are the width and height of the image respectively. According to this, the information entropy of the image is obtained:

[0062]

[0063] 2) Spatial frequency

[0064] The spatial frequency SF of the image is calculated by the spatial row frequency RF and the spatial column frequency CF, that is,

[0065]

[0066]

[0067]

[0068] Wherein m and n are the number of pixels in the width direction and the height direction of the image respectively, and y(i, j) is the gray value of the image in the ith row and the jth column, mainly reflecting the rate of change of the image gray (the color image is converted into a gray image first). The greater the value of the spatial frequency SF, the more complex the image.

[0069] 3) Noise standard deviation

[0070] The noise standard deviation φ is calculated by using an estimation model based on arithmetic average filtering.

[0071] For the second-order difference characteristics of the image, the Laplacian operator is used for convolution and the whole image is averaged, and the specific calculation formula is as follows:

[0072]

[0073] where W and H are the width and height of the image respectively, is the Laplacian operator, and * is the convolution operation. I(X,Y) represents the similarity of two images in brightness information. The greater the value of the noise standard deviation, the more complex the image.

[0074] Therefore, the global complexity of the image is defined as follows:

[0075] C g = α1·H + α2·SF + α3·φ

[0076] where α1, α2, and α3 are the weights of information entropy, spatial frequency, and noise standard deviation, respectively. The weights are initially assigned equally, i.e., each accounts for After that, they are adjusted according to specific circumstances.

[0077] 2. Local complexity

[0078] The local complexity of the image is represented by the difference between the target region and the background region in the image.

[0079] We define the target region as a rectangular region centered on the target with a size of U times the target, and the rest as the background region, where U is the multiple of the entire image region to the target region. The mutual information formula is as follows:

[0080] ΔH = H t + H b - H t,b

[0081] where H t and H b are the information entropy of the target region and the background region, respectively, and H t,b is the joint entropy. The smaller the mutual information, the greater the local complexity, and the more difficult it is to separate the target from the background.

[0082] The contrast difference Δθ of the target region and the background region is calculated by the square root of the sum, and the formula is as follows:

[0083]

[0084] where μ t is the mean gray value of the target region, μ b is the mean gray value of the background region, represents the standard deviation of the gray value of the target region. The smaller the contrast difference Δθ, the greater the local complexity of the image.

[0085] The local complexity is defined as:

[0086] C l = β1·ΔH + β2·Δθ

[0087] Where β1, β2 are the weights of mutual information and contrast difference respectively. The weights are initially assigned equally, i.e. 50% each, and then adjusted according to specific circumstances.

[0088] So far, the scene complexity of the image is defined as:

[0089] C = α·C g + β·C l

[0090] Where α, β are the weights of global complexity and local complexity respectively. The weights are initially assigned equally, i.e. 50% each, and then adjusted according to specific circumstances.

[0091] II. Analysis of server available resources

[0092] The calculation method of server available resources needs to consider multiple factors, including the computing power, bandwidth, memory and storage space of the server.

[0093] First, consider the available proportion of computing power resources I e , bandwidth resources B e , memory resources G e , storage space D e of the edge server in the current state, and calculate the available resource ratio of the edge server by the weighted sum of the above resources as follows:

[0094]

[0095] Second, consider the available proportion of computing power resources I c , bandwidth resources B c , memory resources G c , storage space D c of the cloud server in the current state, and calculate the available resource ratio of the cloud server by the weighted sum of the above resources as follows:

[0096]

[0097] III. Video task collaborative allocation scheme

[0098] Determine two complexity thresholds C1 and C2 (C1 < C2) according to the actual application scenario of the user, divide three complexity levels (simple, medium, complex) based on the two complexity thresholds, and determine whether the current video task should be uploaded to the edge server or the cloud server according to the complexity level.

[0099] When C≤C1, it indicates that the current task is relatively simple and should be uploaded to the edge server for processing; when C≥C2, it indicates that the current task is relatively complex and needs to be uploaded to the cloud server for processing; when C1

[0100] When the task complexity is combined with the available server resource condition for comprehensive analysis, the task coefficient Tc and the available server resource coefficient SRc are defined as follows:

[0101]

[0102]

[0103] When T C ≤SR C , it indicates that the available resource proportion of the current edge server is higher or the complexity of the task is relatively low, and at this time it is suitable to upload the task to the edge server for processing. When T C >SR C , it indicates that the available resource proportion of the current cloud server is higher or the complexity of the task is relatively high, and it is more suitable to upload the task to the cloud server for processing. Thus, the edge-cloud collaborative allocation scheme of the video task is determined through the comprehensive analysis of the scene complexity and the server resource.

[0104] Four, specific implementation steps

[0105] The specific implementation steps of the end-edge-cloud collaborative scheme based on visual scene perception proposed by the application are as follows:

[0106] 1. The scene complexity of the visual data of the Internet of Things intelligent terminal is perceived, specifically including:

[0107] 1) Extract information entropy H, spatial frequency SF and noise standard deviation φ to represent the global complexity C of the image g , wherein,

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] C g = α1·H + α2·SF + α3·φ

[0114] 2) Calculate the mutual information ΔH and the contrast difference Δθ between the target region and the background region in the image, which represent the local complexity C of the image l wherein,

[0115] ΔH = H t + H b - H t,b

[0116]

[0117] C l = β1·ΔH + β2·Δθ

[0118] 3) Calculate the scene complexity C of the image

[0119] C = α·C g + β·C l

[0120] 2. Comprehensive analysis of the available resources in the current system.

[0121] 1) Calculate the available resource proportion of the overall edge server

[0122]

[0123] 2) Calculate the available resource proportion of the overall cloud server

[0124]

[0125] 3. Reasonable allocation of video tasks based on the joint analysis results of scene perception and available resource conditions.

[0126] 1) Determine two complexity thresholds C1 and C2 (C1 < C2) according to the actual application scenarios of the user, and divide three complexity levels based on these two complexity thresholds, and then determine whether the current video task should be uploaded to the edge server or the cloud server.

[0127] When C ≤ C1, upload to the edge server for processing; when C ≥ C2, upload to the cloud server for processing; when C1 < C < C2, comprehensive analysis of the resources of the joint server.

[0128] 2) When C1 < C < C2, according to the scene complexity and the comprehensive judgment of the server resources, determine whether the current video task should be uploaded to the edge server or the cloud server.

[0129] a. Calculate the task coefficient Tc and the server available resource coefficient SRc:

[0130]

[0131]

[0132] b. When T C ≤ SR C , upload the task to the edge server for processing; when T C > SR C , upload the task to the cloud server for processing.

[0133] Through the specific embodiments of the application described above, the end-edge-cloud collaborative method and system based on visual scene perception are proposed. For a given input, the complexity of the task can be preliminarily determined according to visual information collection and perception fusion preprocessing, ensuring that simple tasks can be transmitted to the edge server for processing, and complex tasks can be transmitted to the cloud server for processing, thereby reducing unnecessary data transmission and computing resource consumption. The preprocessing results are comprehensively analyzed in combination with various resource conditions of the server, and the edge-cloud collaborative allocation scheme of the video task is finally determined. Under the driving of diversified video task reasonable allocation, the video task processing efficiency is improved, and the consumption of various resources is effectively reduced, thereby reducing the cost.

[0134] Although the present application has been described in terms of preferred embodiments, it is not intended to be limited to the embodiments described as the present application extends to equivalents of which the scope of each claim limits.

Claims

1. A device-edge-cloud collaboration method based on visual scene perception, wherein the device-edge-cloud is a smart terminal, an edge server, and a cloud server, characterized in that: The method comprises the following steps: S1: Intelligent terminal perceives the complexity of visual scenes; in, In step S1, the scene complexity C is calculated according to the following formula: Among them, C g is the global complexity, C l is the local complexity, , are the weights of global complexity and local complexity respectively; The global complexity C of the image g The calculation method is as follows: Among them, H is the information entropy, SF is the spatial frequency, φ is the noise standard deviation, , , are the weights of information entropy, spatial frequency and noise standard deviation respectively; The local complexity C of the image l The calculation method is as follows: in, is mutual information, For poor contrast, , are the weights of mutual information and contrast difference, respectively; S2: classifying the video task to be processed into a simple video task, a medium video task, and a complex video task according to the complexity of the visual scene; S3: For the simple video task, the smart terminal transmits the video task to the edge server for calculation; for the complex video task, the smart terminal transmits the video task to the cloud server for calculation; and for the medium video task, the smart terminal distributes the video task between the edge server and the cloud server according to the ratio of available resources of the edge server and the cloud server; in, In step S3, the available resource ratio of the edge server is calculated as follows: ， Among them, I e , B e , G e , D e They are the available ratio of computing power resources, bandwidth resources, memory resources, and storage space in the current state of the edge server. The calculation method of the available resource ratio of the cloud server is: ， Among them, I c , B c , G c , D c They are the available ratios of computing power resources, bandwidth resources, memory resources, and storage space in the current state of the cloud server. The method for allocating video tasks is: comparing the task coefficient Tc with the server available resource coefficient SRc, If T C ≤SR C ,The video task is uploaded to the edge server for processing, If T C >SR C , the video task is uploaded to the cloud server for processing, in , C1, C2 are preset thresholds and C1 <C2, .

2. The device-edge-cloud collaboration method based on visual scene perception according to claim 1 is characterized in that: In step S2, the method of dividing the simple video tasks, medium video tasks and complex video tasks is: when the scene complexity is lower than or equal to the preset threshold C1, it is a simple video task; when the scene complexity is higher than or equal to the preset threshold C2, it is a complex video task; when the scene complexity is higher than the preset threshold C1 and lower than the preset threshold C2, it is a medium video task, where C1 is less than C2.

3. The device-edge-cloud collaboration method based on visual scene perception according to claim 1 is characterized in that: The information entropy H is calculated as follows: in, is the occurrence probability of the gray value of the feature binary in the image.

4. The device-edge-cloud collaboration method based on visual scene perception according to claim 1 is characterized in that: The calculation method of the spatial frequency SF is as follows: Where RF is the spatial row frequency, CF is the spatial column frequency, m and n are the number of pixels in the width and height directions of the image respectively, and y(i,j) is the grayscale value of the image in the i-th row and j-th column.

5. The device-edge-cloud collaboration method based on visual scene perception according to claim 1 is characterized in that: The mutual information The calculation method is as follows: Among them H t and H b are the information entropy of the target area and the background area respectively, H t,b is the joint entropy.

6. The device-edge-cloud collaboration method based on visual scene perception according to claim 1 is characterized in that: The contrast difference is calculated as follows: where μ t is the grayscale mean of the target area, μ b is the grayscale mean of the background area, is the grayscale standard deviation of the target area.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the method according to any one of claims 1 to 6.

8. A computing system comprising: a storage device, and one or more processors; The storage device is used to store one or more computer programs, and when the computer programs are executed by the processor, they are used to implement the method according to any one of claims 1 to 6.

Images