Road segment space division-based fusion method for optimizing field of view of vehicle-mounted camera

Abstract

The key problem of building global vision in vehicle networking is that each vehicle can obtain the overall information of its surrounding environment through the interaction between vehicles and edge servers. However, under the current technical conditions of vehicle networking, data transmission and running efficiency are one of the constraints to achieve the construction of global vision. Therefore, this paper designs a method of optimizing and fusing the field of view of vehicle-mounted cameras based on road segment space division, which includes four steps: (1) data preprocessing: grid connected component construction algorithm based on breadth-first and union set optimization (BGBC); (2) initial allocation: connected component-based greedy allocation algorithm (CCFA); (3) optimization: segmented genetic algorithm optimization guided by greedy strategy; (4) performance index verification through comparative experiments. This algorithm can help reduce the time and energy required for a data packet transmission, while reducing the complexity of running the algorithm and reducing the consumption of edge servers.

Description

TECHNICAL FIELD

[0001] The application relates to a vehicle-mounted camera field-of-view optimization fusion method based on road section space division and belongs to the intelligent transportation field. BACKGROUND

[0002] Vehicle networking technology is rapidly developing and has received a lot of attention and investment worldwide. This technology can provide unprecedented convenience, efficiency and safety for our modern transportation. Global vision is a key issue in vehicle networking, mainly referring to the fact that through the interaction between vehicles, each vehicle can obtain the overall information of its surrounding environment. This enables each vehicle to perceive and respond to all situations in the most complete and accurate way, thereby greatly improving the safety and efficiency of driving. The construction of global vision is a core issue in the research of vehicle networking and is of great significance. However, under the current conditions of vehicle networking technology, data transmission and operation efficiency are one of the factors restricting the construction of global vision. Since each vehicle obtains a large amount of information through sensors or cameras, a large amount of network bandwidth and processing capacity is required when transmitting and analyzing data. This may cause high communication delay, data loss and increased complexity, making it difficult to achieve complete global vision. The sensor or camera data collected by the vehicle can be directly sent to the edge server for real-time processing, analyzing vehicle status, driving route and other information, and realizing intelligent driving, vehicle networking, vehicle safety and other applications according to the analysis results, while the edge server can realize intelligent control functions through interaction with the vehicle.

[0003] Based on the above problems and functions, the present application proposes a global vision construction algorithm aimed at reducing data transmission and operation efficiency. This algorithm adopts a vehicle-mounted camera field-of-view optimization fusion method based on road section space division, which can reduce redundant operations in calculation, thereby further optimizing the algorithm to improve communication and compilation speed and accuracy. This algorithm can help reduce the time and energy required for a data packet transmission, while reducing the complexity required for running the algorithm and reducing the complexity of the edge server. These advantages make this algorithm have great potential and expected effect in realizing the construction of global vision. The technical solution of the present application not only can reduce the influence of these problems to a very small extent, but also provides an effective method for better realizing the construction of global vision, and is expected to be widely used in the research and application of vehicle networking technology. SUMMARY

[0004] The problem to be solved by the present application is: for a plurality of vehicles in the signal coverage area of the edge server, as shown in the accompanying drawings, Figure 1As shown, each vehicle is equipped with four cameras to record the visible view, and it is necessary to determine an optimal combination scheme of vehicle-mounted cameras to effectively construct the global view of the entire road section based on the local view pictures uploaded by all cameras in the combination. The visible view distance of the front camera and the rear camera of each vehicle is consistent, and the visible view distance of the left and right cameras is consistent. The view of each camera is defined as a trapezoidal view, and the entire road section space is divided into a plurality of grids of the same size.

[0005] The present application provides a vehicle-mounted camera view optimization fusion method based on road section space division, which includes four core steps: (1) data preprocessing stage: grid connected component construction algorithm based on breadth-first and union set optimization (BGBC); (2) initial allocation stage: connected component-based greedy allocation algorithm (CCFA); (3) optimization stage: greedy strategy guided segmented genetic algorithm optimization; and (4) comparative experiment to verify the performance of the algorithm. The specific steps are as follows:

[0006] 1. Grid connected component construction algorithm based on breadth-first and union set optimization

[0007] Based on the global view of the space division, the present application has obtained a candidate grid set G of each camera of each vehicle l (l = 0, 1, 2...m, m is the number of pre-allocated grids). At this time, all pre-allocated grids are discontinuous and scattered on the entire road. The allocation of grids to cameras is to achieve efficient uploading of the global view and reduce the amount of repeated data and data uploaded. If the candidate camera list of some grids is consistent, the probability of these grids being continuous is high. If some continuous grids are allocated to different cameras, these cameras will all need to upload the view represented by these overlapping grids, which will greatly waste the upload bandwidth and storage space. By allocating some continuous grids to the same camera as much as possible, the possibility of such a large amount of overlap can be maximized to improve data utilization. The present application focuses on the fusion and optimization scheme of the global view. The grid connected components are constructed using the breadth-first algorithm and the union set, which includes two steps: (1) constructing the union set; and (2) breadth-first search to construct the connected components.

[0008] Step 11: Disjoint-set is a data structure and also an algorithm based on the data structure. In the data structure, each element has a parent representing its ancestor, which is equivalent to each element belonging to a set. We can use the data structure to implement set operations, including building sets, merging sets, finding elements, set sizes, etc., to achieve fast lookup and merging of set elements. In this invention, disjoint-set is used to implement the connectivity grouping of data points. My disjoint-set supports union, find, and connect. Find is used to query whether a node belongs to a known connected component group, union operation is used to combine two connected components into one, and connect phase is used to get the specific nodes contained in each connected component.

[0009] Different connected components are independent of each other, and all nodes in the same connected component can reach each other. The parent array is used to represent the parent node of each node, and the size array is used to represent the size of each connected component. When merging nodes, first find the root node of the connected component where the node is located, then if the two nodes belong to the same connected component, do nothing, otherwise merge the smaller connected component into the larger one and update the size information of the root node corresponding to the connected component.

[0010]

[0011] In the disjoint-set, each element has a parent node, and when an element's parent node is itself, it indicates that it is an independent set. I use path compression-based find here, which not only finds the representative element of the set during the find process, but also directly points the parent nodes of all elements on the path to the representative element. The time complexity of this method is usually better than that of the tree-based find method.

[0012]

[0013] Next, we need to get all the node coordinates in the connected component where a certain node is located. We create a connected list to store the node coordinates in the connected component. Then, we iterate through all the nodes and add the node's coordinates to the list if the root node of the connected component where the node is located is rootP. Finally, return the list of all node coordinates connected, which can get all the node coordinates in the connected component where a certain node is located.

[0014] Step 12: Breadth-first search constructs connected components. Using the breadth-first search algorithm in graph theory, each grid is constructed as a node, and the starting node is chosen to be the left lower corner of the entire road. Starting from the starting node, the breadth-first search is performed. During the traversal process, only the nodes represented by the pre-allocated grids are processed, and the nodes with consistent candidate camera lists are constructed into a connected component. During the search process, the neighboring nodes of the current node are traversed. If the node does not belong to the pre-allocated node, it is skipped. Otherwise, from bottom to top and from left to right, when a grid is traversed, it is first determined whether the surrounding grids meet the merging conditions. If they do, they are merged and then added to the queue for further traversal. That is, it is determined whether the candidate camera list of the current node is consistent with that of the previous node. If it is consistent, the current node and the previous node are merged into a connected component. When the last node is traversed, all pre-allocated grids have formed different connected component sets according to the merging conditions, and the candidate camera list of each grid in each connected component set is consistent. The algorithm design is as follows:

[0015]

[0016] 2. Greedy allocation algorithm based on connected components

[0017] The greedy algorithm is used to allocate the connected components as the basic unit. This algorithm is used to calculate the visual field allocation scheme. Because of the data preprocessing, we do not need to judge and allocate all pre-allocated grids at this time. In the case of a large amount of data, the search efficiency is very low. Using the union-find set can reduce the search time efficiency to O(1). In this application scenario, each camera has its own visual field range. Therefore, selecting the camera that can see the most grids is actually selecting the camera that can cover the most grids. The advantage of this is that it can make full use of the visual field range of the camera and allocate as many grids as possible to the camera that can see them. This reduces the number of cameras that need to upload pictures. Operation steps: We only need to process the root node of each connected component and allocate all grids in the connected component to which the root node belongs to the camera in the candidate camera list that can see the most grids. And when allocating connected components, such as the candidate camera list of connected component G1 is "ABCDE" and the candidate camera list of connected component G2 is "ABH". If A is allocated when G1 is allocated, then if H camera has not been allocated before, based on the greedy strategy, it is considered that the camera will not be allocated afterwards. Therefore, camera A is still preferred to be allocated. This further improves the utilization rate of camera A. The algorithm idea is as follows:

[0018]

[0019] 3. Greedy strategy-guided segmented genetic algorithm

[0020] Step 31, a segmented genetic algorithm guided by a greedy strategy. Although a greedy strategy yields an approximate solution, it is a combination of locally optimal strategies and cannot guarantee an optimal solution. In general, it can only obtain a solution close to the optimal one. Therefore, we need to use a genetic algorithm for optimization. The segmented genetic algorithm requires the following steps: (1) setting the encoding method; (2) initializing the population based on the greedy strategy; (3) setting the fitness function; (4) concurrent single-point crossover operation; (5) concurrent mutation operation; (6) roulette wheel selection strategy.

[0021] (1) Encoding Method. Segmented real-number encoding is used. The model's pre-assignment is a grid set, which is then assigned to the corresponding vehicles and cameras. Furthermore, this is a constrained genetic algorithm assignment. Chromosomes are segmented and divided equally. We propose the following gene representation, such as... Figure 2 As shown, if the i-th element is in set G, then the subsequent sets d of the camera and vehicle lists can be obtained according to the order of the grid cells in the set. i An element is selected from the set, which ensures that an element must be assigned to one and only one subset, thus preventing the generation of any genes that violate the constraints. This is well-suited to the present invention.

[0022] (2) Initial Population. The optimal solution obtained in step 2 is used as the input of one chromosome to the initial population of the genetic algorithm. The length of the pre-assigned grid set is m, and segmented real-number encoding is used. In this invention, the chromosome length is m*3, dividing the gene chain into m segments. Each segment corresponds to a pre-assigned grid. The first two digits of each segment correspond to the vehicle number assigned to that grid, and the third digit corresponds to the camera number assigned to that grid. (See attached...) Figure 2 The optimal solution obtained through a random mutation greedy algorithm is used to generate the final initial population. This increased randomness allows the genetic algorithm to find the optimal solution more quickly. The length of each chromosome, i.e., the number of genes, is determined by the total number of computational tasks. Since the population consists of multiple chromosomes, in this invention, the population is represented by an array `pop`; the population size is defined as `popsize`.

[0023] (3) Fitness function. In order to measure the degree of excellence of each individual in the selection process, a fitness function that can directly reflect the performance of the individual is needed. In this invention, because our goal is to assign as few cameras as possible to all pre-allocated fields of view. Therefore, the fitness function is set to the minimum number of cameras finally assigned. The fitness function of the number of different cameras assigned to the chromosome gene chain, the fitness function value becomes the fitness value, the calculation method is as follows:

[0024]

[0025] (4) Concurrent crossover operation. I set the monotonic crossover, single-point crossover: set the crossover probability and the crossover individual, according to the breaking point to carry out chromosome crossover. Example: before crossover: a = <123234 | 081162>, b = <062031 | 182113>; After crossover: a = <123234 | 182113>, b = <062031 | 081162>. The crossover rate of the crossover operation is 80%, and the average is 80% of the chromosomes that are crossed. Because our individuals are segmented, each segment corresponds to a pre-allocated grid, after crossover, we need to verify the crossover operation, which can be carried out under the condition of meeting the constraint condition. Monotonic crossover is based on randomly generated crossover points, and is a relatively simple and effective crossover method in the crossover operation of genetic algorithm. Its basic idea is to preserve the gene sequence information in the parent chromosome as much as possible, while ensuring that each gene still appears only once after crossover. The algorithm idea is as follows:

[0026]

[0027] (5) Mutation operation. The basic idea is: according to the mutation probability to select the mutation site, change the real number bit. When mutation, because there is a constraint between each segment of the individual and the candidate vehicle / camera corresponding to the grid, so only in the candidate camera set of the grid can be mutated. Mutation probability is set to 1%.

[0028] (6) Selection operation. The basic idea is that the probability of a good individual being selected is proportional to its fitness value, i.e. individuals are selected according to a certain rule determined by their fitness values to enter the next generation. First: calculate the fitness value of each individual: for each individual, calculate the corresponding fitness value according to the fitness function. Next: randomly generate the selection probability of each individual: for each individual, calculate the probability of being selected in this round according to its fitness value. The commonly used method is to calculate the selection probability of each individual according to the value of the fitness function. Specifically, the sum of the values of the fitness functions of all individuals is taken as the denominator, and the value of the fitness function of each individual is divided by the sum as the selection probability of the individual.

[0029]

[0030] Then: create a roulette wheel according to the calculated selection probability of each individual. Specifically, divide the random numbers between 0 and 1 into multiple intervals, and the length of each individual's interval is proportional to its selection probability. Finally: select individuals: randomly generate a random number between 0 and 1, and according to which interval it falls into, select the individual to enter the next generation. Repeat this process until the number of selected individuals is the same as the parent generation. In the entire algorithm, the chromosomes in the current generation population are usually sorted according to their fitness values, and some individuals are selected as the parents of the next generation population according to a certain probability. Therefore, in the selection operation, we need to calculate the fitness value of each individual and select some individuals using appropriate methods. Specifically, we first define three arrays evals, p and q to store the fitness values of all chromosomes, selection probabilities and cumulative probabilities. Then, we use a loop to traverse the entire population, calculate the fitness value of each individual, and store it in the evals array. While calculating the fitness value, we also record the optimal solution in the population and the corresponding individual information. Next, we calculate the selection probability and cumulative probability of each chromosome to construct the roulette wheel used for selection. Specifically, we use two loops to traverse the entire p and q arrays. The specific algorithm is as follows:

[0031]

[0032] (7) Number of iterations. Set to 200 times. Experiments have shown that this number of iterations can well obtain good solutions, and if a better solution has not appeared after 20 iterations, the iteration is ended and it is considered that the optimal solution has been found.

[0033] 4. Comparison of algorithm effect verification

[0034] The results show that the optimal solution obtained by the segmented greedy genetic allocation algorithm based on connected components is significantly better than that of the genetic algorithm and the greedy algorithm, and its runtime performance is far lower than that of the genetic algorithm. Even with the most naive allocation scheme, the runtime reaches the second level, and the genetic algorithm can achieve a few milliseconds. However, the greedy algorithm and the segmented greedy genetic algorithm based on connected components can obtain a relatively good solution in a much shorter time. Although the segmented greedy genetic algorithm based on connected components is slightly less time-efficient than the greedy algorithm and the greedy algorithm based on connected components, the time efficiency is still kept within tens of milliseconds. However, the proposed method significantly outperforms the greedy algorithm in obtaining the optimal solution. For example, when the road length is set to 1000, the width to 20, the grid side length to 1, the front-to-back camera field-of-view distance to 16, and the left-to-right camera field-of-view distance to 12, the proposed method requires significantly fewer cameras to upload images compared to the greedy algorithm. For instance, when the number of vehicles is 20, the proposed method reduces the number of assigned cameras by approximately 35% compared to the greedy algorithm. As the number of vehicles gradually increases, for example, to 28, the reduction is approximately 45%. Therefore, experiments have verified the real-time performance and effectiveness of the algorithm, significantly reducing the number of vehicles and images requiring data upload. Attached Figure Description

[0035] Figure 1 Diagram of trapezoidal field of vision for a single vehicle

[0036] Figure 2 A schematic diagram of the segmented genetic algorithm encoding method. Detailed Implementation

[0037] To better explain the content of this invention, an example is given below for specific illustration.

[0038] For details on how to obtain the set of pre-allocated rasters, please refer to the appendix. Figure 1 This can be obtained intuitively. To better illustrate the example, we will set parameters for a small dataset here. The road length is set to 40m, the width to 6m, the grid length to 2m, the grid width to 2m, the number of obstacles to 1, the field of view distance between the front and rear cameras to 6m, and the field of view distance between the left and right cameras to 3m. By randomly placing vehicles on this road, the vehicle positions are as follows:

[0039] Table 1 Vehicle Locations

[0040]

[0041] Following the calculation formula explained above, the set of raster indices corresponding to the trapezoidal field of view of each camera is calculated as follows:

[0042] Table 2 pre-allocated grids

[0043]

[0044]

[0045] Firstly, these pre-allocated grids are constructed into connected components according to the principle of consistent candidate cameras, and are saved in the union-find set. For example, the candidate camera list of grid (1, 1) and (2, 2) are both (7, 1) and (9, 1). It means that both of these two grids can be seen by the 1st camera of the 7th car and the 1st camera of the 9th car. So these two grids are constructed into a connected component, and are put into the union-find set. Then according to the greedy algorithm designed above, each node (grid) in each connected component is allocated to the camera in the candidate camera list which can see the most grids. In this case, (1, 1) and (2, 2) are allocated to the 1st camera of the 7th car, because this camera can see five grids. In this way, all the pre-allocated grids are allocated one by one.

[0046] Then the approximate solution is randomly mutated to generate the initial population of genetic algorithm. The piecewise genetic algorithm is entered to optimize. The final solution is obtained.

[0047] Table 3 final allocation scheme

[0048]

Claims

1. A method for optimizing and fusing the field of view of a vehicle-mounted camera based on road segment spatial division, characterized in that, The method includes the following steps: Step 1: Construct grid connectivity components based on disjoint-set data structure and breadth-first search; Step 11: Establish a list of connected components based on the disjoint-set data structure; A list of relationships between grid nodes and connected components on a road segment is created through search, merge, and connect operations. Step 12: Construct connected components based on breadth-first search; The breadth-first search algorithm in graph theory is adopted. The breadth-first search starts from the bottom left corner grid node of the road segment. During the traversal, only the nodes represented by the pre-assigned grid are processed, and nodes with the same candidate camera list are constructed into a connected component. Step 2: Design a greedy algorithm based on connected components for initial allocation; Using the constructed connected components as the basic allocation unit, the camera with the most grid field of view is selected from the candidate camera list for the representative root node of the connected component, and all grid nodes of the connected component are assigned to the camera. In order to maximize the field of view coverage and image quality, the number of cameras already assigned in each optional camera scheme is calculated to further maximize the utilization of the visible field of view of each camera. Step 3: Design a segmented genetic algorithm guided by a greedy strategy for optimal allocation; Based on the initial allocation scheme obtained in step 2, a genetic algorithm with candidate camera list constraints is designed to complete the optimized allocation of road segment grid nodes; Step 31: Express the allocation scheme as chromosomes using a segmented real number encoding method; Step 32: Generate the initial population of the genetic algorithm by randomly mutating the chromosomes obtained by the initial allocation scheme. The chromosome length is m*3, divided into m segments, each segment corresponds to a grid. The first two digits of each segment correspond to the vehicle number allocated to the grid, and the third digit corresponds to the camera number allocated to the grid. Step 33: Set the fitness function to the number of cameras covered by the allocation scheme; the fewer the better. Step 34: Concurrent crossover operation. Based on the set crossover probability, the crossover individuals are crossbred at randomly generated crossover points. After crossover, it is necessary to verify whether the candidate constraints are still satisfied. Step 35: Mutation operation, select mutation sites according to the set mutation probability and complete the change of real bits based on the corresponding candidate camera set; Step 36: Selection operation, obtain the fitness value of each chromosome and calculate its selection probability and cumulative probability accordingly, so as to construct a roulette wheel to complete the selection of individuals entering the next generation of iterations; Step 37: Repeat steps 34 to 36 to perform population optimization iterations until the set number of iterations is reached. The solution obtained at the end of the iteration is used as the final allocation scheme.

Images