Multi-vision task processing system based on edge cloud collaboration

Abstract

The invention provides a multi-visual task processing system based on edge cloud collaboration, the system is used for receiving and processing a plurality of visual tasks sent by a plurality of terminals, a dynamic feature extraction framework is introduced to the edge of the system, and the visual tasks are extracted according to the types of the visual tasks. The adaptive feature extraction model is matched from the pre-configured feature model library, so that the problem of information redundancy or loss possibly occurring in a traditional feature extraction mode is avoided, and the pertinence and efficiency of preprocessing are greatly improved. Aiming at the problem of poor adaptability of resource allocation, the invention designs a self-adaptive resource allocation framework, and the framework flexibly adjusts a resource allocation strategy according to resource information of side equipment and cloud equipment, so that the system operation efficiency, stability and reliability are effectively improved, and the adaptability of the system to a complex scene for processing a plurality of heterogeneous visual tasks is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of edge-cloud collaboration, specifically to the field of machine learning model reasoning, and more specifically to a multi-visual task processing system based on edge-cloud collaboration. Background Art

[0002] As digitalization and intelligence develop rapidly, machine vision technology has great potential for application in many fields such as industrial manufacturing and intelligent security, such as industrial target detection and instance segmentation, and key point detection of security personnel. Society's demand for it is rising, and higher performance and efficiency are required. At the same time, facing the surge in data volume and the complexity of scenes, improving the ability to process visual tasks under limited resources is a key challenge for scientific and technological progress.

[0003] At present, visual task processing technology mainly adopts two modes: cloud computing and edge-cloud collaboration. However, both have significant limitations in multi-task processing scenarios. The cloud computing model has low resource utilization efficiency when processing multiple visual tasks. This method directly uploads the original image to the cloud for processing, and fails to fully consider the differentiated resource requirements of different tasks, resulting in a waste of cloud device bandwidth and edge server computing resources. At the same time, its task adaptability is poor, and the unified processing flow cannot be optimized according to the characteristics of different visual tasks, thus affecting the overall performance. In addition, the cloud computing model fails to fully utilize the resource potential of the edge side and is prone to large delays in wide area network transmission. To make up for the shortcomings of cloud computing, the edge-cloud collaboration model came into being. This model optimizes processing efficiency by sinking some computing tasks to the edge. However, the edge-cloud collaboration model also has some problems that need to be solved. First, its preprocessing scheme is relatively fixed, and the edge uses a unified feature extraction method, which fails to perform differentiated processing for the feature requirements of different tasks, which may lead to information loss or redundancy. Secondly, the resource allocation mechanism lacks flexibility and cannot dynamically adjust the resource allocation strategy according to the task type and system status, making it difficult to adapt to complex and changing application scenarios. Finally, the coordination between tasks is weak, and each task is processed independently, failing to fully utilize the information sharing and collaborative optimization opportunities between tasks, limiting the improvement of overall performance. In the field of machine vision, tasks such as target detection, instance segmentation, and key point detection of people show significantly different sensitivity and demand characteristics in terms of resource dimensions, such as bandwidth and feature extraction models.

[0004] To sum up, at the feature extraction level, the current visual task processing mode based on edge-cloud collaboration adopts a fixed and unified feature extraction method at the edge, which fails to meet the unique feature requirements of different tasks, resulting in the loss of feature information, and thus having a negative impact on the subsequent task processing results. In terms of resource allocation, due to the lack of the ability to dynamically adjust the allocation strategy based on the task type and the real-time status of the system, it is difficult to adapt to complex and changeable application scenarios, which not only causes unreasonable resource utilization, but also leads to the problem of low efficiency in visual task processing. Therefore, there is an urgent need for a multi-visual task processing solution that can intelligently and collaboratively optimize the resource allocation of multiple visual tasks to achieve a dual improvement in visual task processing efficiency and resource utilization efficiency.

[0005] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0006] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-visual task processing system based on edge-cloud collaboration.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a multi-visual task processing system based on edge-cloud collaboration is proposed, wherein the system receives multiple visual tasks sent by multiple terminals and processes them, wherein the system includes an edge device and a cloud device, wherein the edge device is used to receive multiple visual tasks and perform feature extraction processing and upload the feature extraction results to the cloud, wherein the cloud device is used to perform visual task reasoning based on the feature extraction results uploaded by the edge device, wherein the edge device includes a decision module, multiple edge servers, and a task scheduling module, wherein each edge server is configured with multiple feature extraction models, and each feature extraction model is suitable for feature extraction processing of different visual tasks, wherein the decision module is used to obtain a task scheduling strategy and a task bandwidth allocation strategy that meet a preset processing delay and a preset reasoning accuracy for all visual tasks in a preset manner, wherein the task scheduling module schedules each visual task to a corresponding edge server for feature extraction processing based on the task scheduling strategy, wherein each edge server is used to select a corresponding feature extraction model for feature extraction processing according to the type of the scheduled visual task; wherein the cloud device is used to allocate corresponding bandwidth to each edge server based on the task bandwidth allocation strategy, wherein the bandwidth allocated to each edge server is the sum of the bandwidths of all visual tasks scheduled on the edge server.

[0009] Preferably, the preset method includes: step S1, randomly generating multiple individuals within a predefined range to construct an initial population, wherein each individual includes a task scheduling strategy and a task bandwidth allocation strategy, and using a genetic algorithm to perform multiple rounds of iterations on the initial population to obtain a final population; step S2, determining the individual in the final population that best meets the fitness constraint based on a preset fitness function, and using the individual as the optimal individual; step S3, taking the optimal individual as the center, determining a search area based on a preset search radius, and randomly initializing multiple particles in the search area to construct an initial particle swarm, wherein each particle includes a task scheduling strategy and a task bandwidth allocation strategy, and using a particle swarm optimization algorithm to perform multiple rounds of iterations on the initial particle swarm in the search area until a termination condition is met to obtain a final particle swarm; step S4, determining the particle in the final particle swarm that best meets the fitness constraint based on a preset fitness function, and using the task scheduling strategy and task bandwidth allocation strategy contained in the particle as the final task scheduling strategy and task bandwidth allocation strategy.

[0010] Preferably, in step S1, each round of iteration includes: step S11, using a preset fitness function to determine the fitness of all individuals in the new population formed in the previous round of iteration; step S12, selecting the individual that best meets the fitness constraint as the elite individual; step S13, using a roulette wheel method to select multiple individuals from the new population formed in the previous round of iteration, and performing a crossover operation or a mutation operation on each selected individual to generate multiple new individuals; step S14, forming a new population by combining the elite individual and all the generated new individuals.

[0011] Preferably, in step S13, a crossover operation or a mutation operation is performed on all selected individuals in the following manner: all selected individuals are divided into two sets according to a preset ratio, and the individuals in the two sets are respectively subjected to a crossover operation and a mutation operation to obtain a plurality of new individuals.

[0012] Preferably, in step S3, each round of iteration includes: step S31, using a preset speed update function to determine the speed of each particle in the current iteration round; step S32, based on the speed of each particle in the current iteration round, using a preset position update function to obtain and determine the position of each particle in the current iteration round and update each particle accordingly; step S33, correcting the task scheduling strategy of the updated particles so that each task is only scheduled to one edge server; step S34, when the total bandwidth allocated by the task bandwidth allocation strategy of the updated particles exceeds the total bandwidth of the cloud server, randomly reducing the bandwidth allocated to some visual tasks in the task bandwidth allocation strategy until the total bandwidth allocated by the task bandwidth allocation strategy is less than or equal to the total bandwidth of the cloud server.

[0013] Preferably, the preset speed update function is configured as:

[0014]

[0015] in, represents the iteration round, Represents particles In the The speed of round iterations, Represents particles In the The speed of round iterations, represents the inertia weight, Represents particles In the The distance between the position of the round iteration and the position with the minimum fitness in all previous rounds, Represents particles In the The distance between the position of the round iteration and the position where the fitness of the entire particle swarm is the minimum in all previous rounds of iterations, , represents the acceleration constant, , Represents a random number between 0 and 1.

[0016] Preferably, the preset location update function is configured as:

[0017]

[0018] in, Represents particles In the The position of the round iteration, Represents particles In the The position of the round iteration, Represents particles In the The speed of round iterations.

[0019] Preferably, the termination condition is considered to be met if any one of the following conditions is met: Condition 1, the number of iterations reaches a preset number of iterations; Condition 2, the difference between the fitness of the particle with the smallest fitness in the current iteration round and its fitness in the previous iteration round is less than a preset threshold, wherein the fitness of each particle is determined based on a preset fitness function.

[0020] Preferably, the preset fitness function is configured as:

[0021]

[0022] in, Represents an individual or particle, express The fitness of For The sum of bandwidth allocated to the visual task in a time slot, is the penalty coefficient, Indicates the computing resources occupied by the edge server, yes The number of violations of preset constraints by the corresponding task scheduling strategy and task bandwidth allocation strategy, and is the weight coefficient, and The value of satisfies , where the smaller the fitness of an individual or particle is, the more it complies with the fitness constraint.

[0023] Preferably, the preset constraints include: any visual task can only be scheduled to one edge server, and only one feature extraction model on the edge server is used for feature extraction; the sum of the bandwidth allocated to all visual tasks cannot exceed the total bandwidth of the cloud server; the actual processing delay of each visual task does not exceed the preset processing delay of the visual task; the actual reasoning accuracy of each visual task is not lower than the preset reasoning accuracy of the visual task.

[0024] Compared with the prior art, the advantages of the present invention are:

[0025] According to one embodiment of the present invention, the present invention improves the existing edge-cloud collaborative multi-tasking system. A dynamic feature extraction framework is introduced on the edge side, and an adapted feature extraction model is matched from a pre-configured feature model library according to the type of visual task, so as to avoid the information redundancy or loss problems that may occur in traditional feature extraction methods, greatly improving the pertinence and efficiency of preprocessing. In addition, in order to address the problem of poor adaptability of resource allocation, the present invention designs an adaptive resource allocation framework, which flexibly adjusts the resource allocation policy according to the resource information of the edge device and the cloud device, effectively improving the system's operating efficiency, stability and reliability, and enhancing the system's adaptability to complex scenarios that process multiple heterogeneous visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0027] Figure 1 Schematic diagram of a multi-vision task processing system based on edge-cloud collaboration according to an embodiment of the present invention;

[0028] Figure 2 FIG. 4 is a schematic diagram of an edge server according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] As mentioned in the background technology section, the current multi-visual task processing solution based on edge-cloud collaboration uses a fixed and unified feature extraction method at the feature extraction level, which fails to meet the unique feature requirements of different tasks, resulting in the loss of feature information, and thus having a negative impact on the subsequent task processing results. In terms of resource allocation, due to the lack of the ability to dynamically adjust the allocation strategy based on the task type and the real-time status of the system, it is difficult to adapt to complex and changing application scenarios, which not only causes unreasonable resource utilization, but also leads to the problem of low efficiency in visual task processing.

[0031] To solve the above problems, the present invention proposes a multi-visual task processing solution based on edge-cloud collaboration. This solution optimizes the edge devices, introduces a dynamic feature extraction framework and an adaptive resource allocation framework, accurately identifies the differences in resource requirements of different visual tasks, and adjusts task scheduling and resource allocation in real time, thereby greatly compressing the amount of data transmitted to the cloud, thereby significantly saving network bandwidth and improving transmission efficiency. In addition, through intelligent task scheduling and resource allocation, computing resources can be utilized more efficiently, thereby effectively solving the problem of adaptive resource allocation when multiple heterogeneous visual tasks are executed concurrently.

[0032] According to one embodiment of the present invention, the multi-visual task processing system based on edge-cloud collaboration proposed by the present invention is used to receive and process multiple visual tasks sent by multiple terminals to obtain processing results of the visual tasks, and feed back the processing results to the corresponding terminals, such as Figure 1 As shown in the figure, the multi-visual task processing system based on edge-cloud collaboration includes edge devices and cloud devices, wherein the edge devices are used to receive multiple visual tasks and perform feature extraction processing and upload the feature extraction results to the cloud, and the cloud devices are used to perform visual task reasoning based on the feature extraction results uploaded by the edge devices.

[0033] In order to more clearly illustrate the content of the present invention, the following will be described in detail from two aspects: side devices and cloud devices.

[0034] 1. Side equipment

[0035] According to one embodiment of the present invention, in the present invention, the edge device is mainly composed of a decision module, multiple edge servers and a task scheduling module. Each edge server is equipped with multiple feature extraction models, and each feature extraction model is suitable for feature extraction processing of different types of visual tasks. The decision module plays a key role in policy formulation in the entire system, and obtains task scheduling strategies and task bandwidth allocation strategies based on genetic algorithms and particle swarm optimization algorithms. The task scheduling module accurately schedules each visual task to the corresponding edge server based on the task scheduling strategy obtained by the decision module. After receiving the scheduled visual task, the edge server can quickly select a matching model from the multiple feature extraction models configured by itself according to the specific type of the task, and then carry out feature extraction processing. This precise task scheduling and model matching mechanism effectively avoids the problems of lack of pertinence in feature extraction and unreasonable resource allocation in the existing edge-cloud collaboration mode, and greatly improves the efficiency and quality of visual task processing.

[0036] In order to more clearly illustrate the edge device in the present invention, it will be described from three aspects: decision module, task scheduling module and edge server.

[0037] 1. Decision-making module

[0038] According to one embodiment of the present invention, in the present invention, the decision module uses a search method combining genetic algorithm and particle swarm optimization algorithm to obtain a task scheduling strategy and task bandwidth allocation strategy that meet the pre-configured processing delay and reasoning accuracy of each visual task based on the resource status of the edge server and the bandwidth resources of the cloud server. Specifically, the decision module first uses a genetic algorithm to simulate natural selection and genetic mechanism, uses the global search capability of the genetic algorithm to search for the intermediate optimal task scheduling strategy and task bandwidth allocation strategy, and then uses the particle swarm optimization algorithm to search for local optimal solutions based on the intermediate optimal task scheduling strategy and task bandwidth allocation strategy to further optimize the task scheduling strategy and task bandwidth allocation strategy. The present invention combines the global search capability of the genetic algorithm and the local fine search capability of the particle swarm optimization algorithm to achieve optimal scheduling of visual tasks and reasonable allocation of cloud server resources, effectively improve the overall performance of the system, and overcome the problems of unreasonable resource allocation and low task scheduling efficiency in the traditional edge-cloud collaboration mode.

[0039] According to an embodiment of the present invention, in the present invention, the steps of using a genetic algorithm to obtain an intermediate optimal task scheduling strategy and a task bandwidth allocation strategy are as follows: Step M1, input parameters, including: (1) time slot set: ,in, Indicates the total number of time slots; (2) A collection of visual tasks in a time slot: ,in, express The total number of visual tasks in a time slot; (3) The set of edge servers: ,in, represents the total number of edge servers; (4) the completion delay of each visual task: ; (5) Reasoning accuracy of each task: ; (6) A set of feature extraction models available in all edge servers: ,in, Indicates the total number of feature extraction models that can be used; Step M2, initialize the step size and search area, and define the step size of each axis and , define the search area as ; Step M3, in the predefined search area In step length and The genetic algorithm (GA) population is randomly initialized and the initialized population is ensured to meet the preset constraints. Each individual in the population contains a task scheduling strategy and a task bandwidth allocation strategy. The task scheduling strategy is ,express Task distribution to edge servers Using feature extraction model The binary decision variable, the task bandwidth allocation strategy is , indicating that the cloud server is assigned to the task network bandwidth; step M4, execute the iterative operation of the genetic algorithm until the preset number of iterations of the genetic algorithm is reached, wherein the preset number of iterations of the genetic algorithm can be determined by comprehensively considering the number of visual tasks, the computing resources of the side devices, the complexity of the visual tasks, and the requirements for the accuracy of the results. Each round of iteration includes: firstly using the preset fitness function to determine the fitness of each individual, and selecting the individual that best meets the fitness constraint as the elite individual, and then using the roulette method to select multiple individuals from the new population formed in the previous round of iterations, and performing a crossover operation or a mutation operation on each selected individual to generate multiple new individuals; step M5, using the preset fitness function to determine the fitness of each individual in the population generated by the last round of iteration in step M4, and selecting the individual that best meets the fitness constraint as the intermediate optimal task scheduling strategy and task bandwidth allocation strategy.

[0040] According to one embodiment of the present invention, in step M4 of the present invention, the specific method of selecting individuals by roulette is as follows: First, the selection probability of other individuals except the elite individual is configured, wherein the selection probability of each individual is , is the preset fitness function, is the number of individuals in the population, then, a number of random numbers in the interval [0,1] are randomly generated, and individuals with the closest selection probabilities to each random number are found to select multiple individuals. The roulette wheel is used to select individuals. On the one hand, since individuals that meet the fitness constraints occupy a larger area on the roulette wheel, the possibility of being selected is significantly increased, thereby ensuring the quality of the selected individuals and facilitating the evolution of the algorithm towards a better solution. On the other hand, the randomness of the roulette wheel selection method is reflected in the uncertainty of the pointer pointing to the roulette wheel area, which avoids the algorithm from falling into a local optimal solution too early, maintains the diversity of the population, and enables the algorithm to search in a wider solution space.

[0041] According to one embodiment of the present invention, in the present invention, the individuals selected in step M4 are subjected to crossover or mutation operations to form new individuals. It should be noted that the crossover operation is usually used to explore new solutions, while the mutation operation is to introduce randomness to prevent the algorithm from falling into a local optimum. In the present invention, the specific method of selecting a crossover operation or a mutation operation to process an individual is: the probability of selecting a crossover operation and the probability of selecting a mutation operation are pre-configured to determine the proportion of individuals to be processed by a crossover operation or a mutation operation (the sum of the probability of a crossover operation and the probability of selecting a mutation operation is 1), wherein the probability of a crossover operation is usually large, and the probability of a crossover operation (pc) is usually in the range of 0.6-0.9, for example, when pc=0.8, it means that 80% of the individuals in the population undergo a crossover operation; the probability of a crossover operation is usually small, and the probability of a mutation operation (pm) is usually in the range of 0.01-0.1, for example, when pm=0.05, it means that 5% of the individuals in the population undergo a mutation operation. Selecting crossover or mutation operations to process individuals based on preset selection probabilities can not only balance global and local searches and improve the efficiency of the algorithm in finding the optimal solution, but also introduce new genes and maintain population diversity.

[0042] According to an embodiment of the present invention, when a crossover operation is used to process individuals, a single-point crossover is performed on the task scheduling strategy, and an arithmetic crossover is performed on the task bandwidth allocation strategy. In order to more clearly illustrate the process of using a crossover operation to process individuals, an explanation will be given below in conjunction with Example 1.

[0043] Example 1

[0044] When the task scheduling strategy of individual 1 is x1=[1,0,1,0,1], the task scheduling strategy of individual 2 is x2=[0,1,0,1,0], and the crossover point is the fourth position, the task scheduling strategy of the offspring individual generated after individual 1 performs the crossover operation is: x1′=[1,0,1,1,0], and the task scheduling strategy of the offspring individual generated after individual 2 performs the crossover operation is x2′=[0,1,0,0,1]; if the task bandwidth allocation strategy of individual 1 is w1=[5,10], and the task bandwidth allocation strategy of individual 2 is w2=[8,12], the arithmetic crossover based on weighted average is used to generate offspring, where the arithmetic crossover based on weighted average is , For The random coefficients in and represents the parent individual for the crossover operation, Represents the offspring after crossover. The offspring after crossover of w1 and w2 is [6.5,11].

[0045] According to an embodiment of the present invention, when a mutation operation is used to process an individual, the task scheduling strategy is randomly reversed and a random perturbation is added to the task bandwidth allocation strategy. In order to more clearly illustrate the process of using a mutation operation to process an individual, it will be described below in conjunction with Example 2.

[0046] Example 2

[0047] If the individual's task scheduling strategy is x=[1,0,1,0], randomly reverse some bits in x, when the second bit is reversed, the new offspring individual obtained is x′=[1,1,1,0], when the second and third bits are reversed, the new offspring individual obtained is x′=[1,1,,0], other cases are not listed here one by one; if the individual's task bandwidth allocation strategy is w1=[5,10], add a disturbance value to the task bandwidth allocation strategy, when the disturbance value is -1, the generated new individual is w1′=[4,9], when the disturbance value is 1, the generated new individual is w1′=[4,11]. It should be understood that the disturbance values ​​listed above are only illustrative and not exhaustive.

[0048] According to one embodiment of the present invention, in the present invention, an individual with better performance obtained based on a genetic algorithm is taken as the core, and a search range is determined according to a preset search radius, wherein the preset search radius is determined by constraining the feasible solution space to ensure that the neighborhood of the search radius is within the range that satisfies all preset constraints. A particle swarm optimization algorithm is used within the range to perform a more refined search, thereby avoiding the problem of wasting computing resources due to a large search range, or failing to find a global optimal solution due to a small search range. The use of a particle swarm optimization algorithm to search for the final task scheduling strategy and task bandwidth allocation strategy includes the following steps: randomly initializing multiple particles in the search area, forming a primary particle swarm with all the initialized particles, wherein each particle contains a task scheduling strategy and a task bandwidth allocation strategy, and then using a particle swarm optimization algorithm to perform multiple rounds of iterations on the primary particle swarm in the search area until the termination condition is met to obtain the final particle swarm, and finally, based on the preset fitness function, selecting the particle that best meets the fitness constraint, and using the task scheduling strategy and task bandwidth allocation strategy corresponding to the particle as the task scheduling strategy and task bandwidth allocation strategy, wherein each round of iteration includes : A preset speed update function is used to determine the speed of each particle in the current iteration round; based on the speed of each particle in the current iteration round, a preset position update function is used to obtain the position of each particle in the current iteration round and update each particle accordingly; the task scheduling strategy of the updated particles is corrected so that each task is only scheduled to one edge server; when the total bandwidth allocated by the task bandwidth allocation strategy of the updated particles exceeds the total bandwidth of the cloud server, the bandwidth allocated to some visual tasks in the task bandwidth allocation strategy is randomly reduced until the total bandwidth allocated by the task bandwidth allocation strategy is less than or equal to the total bandwidth of the cloud server, ensuring the feasibility and rationality of bandwidth allocation.

[0049] According to one embodiment of the present invention, in the present invention, the task scheduling strategy of the updated particles is adjusted in the following manner: the Sigmoid function is used to map the values ​​of all elements of the matrix corresponding to the task scheduling strategy of each particle after the update to the interval [0,1], the maximum value of each row of elements in the matrix is ​​set to 1, and the remaining elements are set to 0. For example, if the task scheduling strategy is x=[0.3,0.6,0.4], the corrected task scheduling strategy is x=[0,1,0]. In the iterative process of the particle swarm optimization algorithm, the present invention corrects the task scheduling strategy corresponding to each round of updated particles. This operation effectively avoids the deviation of the solution space caused by the deviation of the scheduling strategy, greatly improves the accuracy of the search solution, and ensures that the algorithm can accurately locate the optimal solution that meets the task requirements and resource allocation.

[0050] According to one embodiment of the present invention, during the cyclic iteration process of the particle swarm optimization algorithm, any one of the following conditions is considered to be satisfied as the termination condition: Condition 1, the number of iterations reaches the preset number of iterations of the particle swarm optimization algorithm, wherein the preset number of iterations of the particle swarm optimization algorithm; Condition 2, the difference between the fitness of the particle with the smallest fitness in the current iteration round and its fitness in the previous iteration round is less than a preset threshold, wherein the fitness of each particle is determined based on a preset fitness function, and preferably, the preset threshold value ranges from 10 -6 -10 -4 .

[0051] According to one embodiment of the present invention, the preset location update function is: , in, represents the iteration round, Represents particles In the The speed of round iterations, Represents particles In the The speed of round iterations, represents the inertia weight, Represents particles In the The distance between the position of the round iteration and the position with the minimum fitness in all previous rounds, Represents particles In the The distance between the position of the round iteration and the position where the fitness of the entire particle swarm is the minimum in all previous rounds of iterations, , represents the acceleration constant, , Represents a random number between 0 and 1.

[0052] According to one embodiment of the present invention, the preset location update function is: ,in, Represents particles In the The position of the round iteration, Represents particles In the The position of the round iteration, Represents particles In the The speed of round iterations.

[0053] According to one embodiment of the present invention, the preset constraint conditions include: (1) , indicating that each task can only be assigned to one feature extraction model on one edge server; (2) , indicating that the processing delay of each task does not exceed its delay requirement; (3) , indicating that the processing delay of each task does not exceed its delay requirement; (4) , indicating that the total bandwidth allocated to the task does not exceed the total bandwidth of the cloud server.

[0054] According to one embodiment of the present invention, in the present invention, the preset fitness function is configured as:

[0055]

[0056] in, Represents an individual or particle, express The fitness of For The sum of bandwidth allocated to the visual task in a time slot, is the penalty coefficient, represents the computing resources occupied by the edge server, which is the sum of the computing resources occupied by all visual tasks on the edge server. yes The number of violations of preset constraints by the corresponding task scheduling strategy and task bandwidth allocation strategy, and is the weight coefficient, and The value of satisfies , where the smaller the fitness of an individual or particle is, the more it complies with the fitness constraint.

[0057] 2. Task scheduling module

[0058] According to an embodiment of the present invention, the task scheduling module schedules the visual task to the designated edge server according to the task scheduling strategy generated by the decision module, so that the edge server provides feature extraction service for the scheduled visual task.

[0059] 3. Edge Server

[0060] According to an embodiment of the present invention, in the present invention, each edge server is pre-configured with a plurality of trained feature extraction models suitable for different types of visual tasks. At the same time, transfer learning technology is introduced to accelerate the feature extraction process by using the feature extraction models trained in the edge server to maintain a high accuracy of feature extraction. Figure 2As shown in the figure, three feature extraction models configured in the edge server are shown, one of which is the Faster R-CNN model suitable for target detection visual tasks, the second is the R-CNN model suitable for instance segmentation visual tasks, and the third is the Keypoints R-CNN model suitable for key point detection visual tasks. It should be understood that the above three feature extraction models are only exemplary and not exhaustive. Other models that can provide feature extraction services for visual tasks can be pre-configured in the edge server. The present invention introduces a dynamic feature extraction framework on each edge server, so that the edge server can automatically select an adaptive feature extraction model according to the scheduled task type to adapt to different types of visual tasks.

[0061] According to one embodiment of the present invention, each edge server provides feature extraction services for one or more machine vision tasks scheduled by a task scheduling module. First, the edge server selects a feature extraction model adapted to the visual task according to the type of the scheduled visual task to provide the corresponding feature extraction service, thereby obtaining feature data of the scheduled visual task. Subsequently, the edge server uploads the obtained feature data to the cloud device so that the cloud device completes the reasoning step of the visual task. In this process, the bandwidth used by each edge server to upload feature data is allocated based on the task bandwidth allocation strategy, and the bandwidth allocated to each edge server is the sum of the bandwidths of all visual tasks scheduled on the edge server.

[0062] 2. Cloud Devices

[0063] According to one embodiment of the present invention, the cloud device allocates corresponding bandwidth resources to each edge server based on the task bandwidth allocation strategy. The bandwidth allocated to each edge server is the sum of the bandwidths of all visual tasks scheduled on it. In addition, the cloud device is also used to provide corresponding model inference services for each visual task based on the feature data uploaded by the edge server to complete the visual task reasoning work. When the visual task reasoning process is completed, the cloud device feeds back the reasoning result of the visual task to the terminal device that generates the visual task. Subsequently, the cloud device optimizes the reasoning result based on the relevant parameters returned by the terminal device to improve the accuracy and reliability of the reasoning result, thereby meeting the complex and changeable actual application needs.

[0064] According to one embodiment of the present invention, the present invention improves the existing edge-cloud collaborative multi-tasking system. A dynamic feature extraction framework is introduced on the edge side, and an adapted feature extraction model is matched from a pre-configured feature model library according to the type of visual task, so as to avoid the information redundancy or loss problems that may occur in traditional feature extraction methods, greatly improving the pertinence and efficiency of preprocessing. In addition, in order to address the problem of poor adaptability of resource allocation, the present invention designs an adaptive resource allocation framework, which flexibly adjusts the resource allocation policy according to the resource information of the edge device and the cloud device, effectively improving the system's operating efficiency, stability and reliability, and enhancing the system's adaptability to complex scenarios that process multiple heterogeneous visual tasks.

[0065] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0066] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multi-visual task processing system based on edge-cloud collaboration, the system receives multiple visual tasks sent by multiple terminals and processes them, the system includes edge devices and cloud devices, wherein: The edge device is used to receive multiple visual tasks and perform feature extraction processing and upload the feature extraction results to the cloud. The cloud device is used to perform visual task reasoning based on the feature extraction results uploaded by the edge device, and is characterized in that: The edge device includes a decision module, multiple edge servers, and a task scheduling module. Each edge server is configured with multiple feature extraction models, and each feature extraction model is suitable for feature extraction processing of different visual tasks. The decision module is used to obtain a task scheduling strategy and a task bandwidth allocation strategy that meet a preset processing delay and a preset reasoning accuracy for all visual tasks in a preset manner. The task scheduling module schedules each visual task to a corresponding edge server for feature extraction processing based on the task scheduling strategy. Each edge server is used to select a corresponding feature extraction model for feature extraction processing according to the type of the scheduled visual task. The cloud device is used to allocate corresponding bandwidth to each edge server based on the task bandwidth allocation strategy, wherein the bandwidth allocated to each edge server is the sum of bandwidths of all visual tasks scheduled on the edge server.

2. The system according to claim 1, characterized in that The preset methods include: Step S1, randomly generating multiple individuals within a predefined range to construct an initial population, wherein each individual includes a task scheduling strategy and a task bandwidth allocation strategy, and using a genetic algorithm to perform multiple rounds of iterations on the initial population to obtain a final population; Step S2, determining the individual in the final population that best meets the fitness constraint based on a preset fitness function, and taking the individual as the optimal individual; Step S3, taking the optimal individual as the center, determining the search area based on a preset search radius, and randomly initializing multiple particles in the search area to construct an initial generation particle swarm, wherein each particle includes a task scheduling strategy and a task bandwidth allocation strategy, and using a particle swarm optimization algorithm to perform multiple rounds of iterations on the initial generation particle swarm in the search area until the termination condition is met, so as to obtain a final particle swarm; Step S4, based on the preset fitness function, determine the particle in the final particle swarm that best meets the fitness constraint, and use the task scheduling strategy and task bandwidth allocation strategy included in the particle as the final task scheduling strategy and task bandwidth allocation strategy.

3. The system according to claim 2, characterized in that In step S1, each iteration includes: Step S11, using a preset fitness function to determine the fitness of all individuals in the new population formed in the previous round of iteration; Step S12, selecting the individual that best meets the fitness constraint as the elite individual; Step S13, selecting multiple individuals from the new population formed in the previous round of iterations by using a roulette wheel method, and performing a crossover operation or a mutation operation on each of the selected individuals to generate multiple new individuals; Step S14, the elite individuals and all the generated new individuals are combined into a new population.

4. The system according to claim 3, characterized in that In step S13, a crossover operation or a mutation operation is performed on all selected individuals in the following manner: All selected individuals are divided into two sets according to a preset ratio, and the individuals in the two sets are subjected to crossover and mutation operations respectively to obtain multiple new individuals.

5. The system according to claim 2, characterized in that In step S3, each iteration includes: Step S31, using a preset speed update function to determine the speed of each particle in the current iteration round; Step S32, based on the speed of each particle in the current iteration round, a preset position update function is used to obtain and determine the position of each particle in the current iteration round and update each particle accordingly; Step S33, modifying the task scheduling strategy of the updated particle so that each task is only scheduled to one edge server; Step S34, when the total bandwidth allocated by the updated particle task bandwidth allocation strategy exceeds the total bandwidth of the cloud server, randomly reduce the bandwidth allocated to some visual tasks in the task bandwidth allocation strategy until the total bandwidth allocated by the task bandwidth allocation strategy is less than or equal to the total bandwidth of the cloud server.

6. The system according to claim 5, characterized in that The preset speed update function is configured as:

7. Among them, represents the iteration round, Represents particles In the The speed of round iterations, Represents particles In the The speed of round iterations, represents the inertia weight, Represents particles In the The distance between the position of the round iteration and the position with the minimum fitness in all previous rounds, Represents particles In the The distance between the position of the round iteration and the position where the fitness of the entire particle swarm is the minimum in all previous rounds of iterations, , represents the acceleration constant, , Represents a random number between 0 and 1.

8. The system according to claim 5, characterized in that The preset location update function is configured as follows:

9. Among them, Represents particles In the The position of the round iteration, Represents particles In the The position of the round iteration, Represents particles In the The speed of round iterations.

10. The system according to claim 2, characterized in that The termination condition is deemed to be met if any of the following conditions are met: Condition 1: The number of iterations reaches the preset number of iterations; Condition 2: The difference between the fitness of the particle with the smallest fitness in the current iteration round and its fitness in the previous iteration round is less than a preset threshold, wherein the fitness of each particle is determined based on a preset fitness function.

11. The system according to any one of claims 2, 3 and 8, characterized in that: The preset fitness function is configured as:

12. Among them, Represents an individual or particle, express The fitness of For The sum of bandwidth allocated to the visual task in a time slot, is the penalty coefficient, Indicates the computing resources occupied by the edge server, yes The number of violations of preset constraints by the corresponding task scheduling strategy and task bandwidth allocation strategy, and is the weight coefficient, and The value of satisfies , where the smaller the fitness of an individual or particle is, the more it complies with the fitness constraint.

13. The system according to claim 9, characterized in that The preset constraints include: Any visual task can only be scheduled to one edge server, and only one feature extraction model on the edge server is used for feature extraction; The sum of bandwidth allocated to all vision tasks cannot exceed the total bandwidth of the cloud server; The actual processing delay of each visual task does not exceed the preset processing delay of the visual task; The actual reasoning accuracy of each visual task is not lower than the preset reasoning accuracy of the visual task.

Images