Depth information real-time generation method based on integrated 3D perception camera module

Abstract

The invention discloses a depth information real-time generation method based on an integrated 3D perception camera module, and the method comprises the following steps: S1, collecting multi-source time sequence data, carrying out the preprocessing, and generating a time sequence data set; s2, constructing a deep residual network, extracting time sequence data features, and generating depth features; s3, introducing a lion group optimization algorithm, and optimizing hyper-parameters of the deep residual network; s4, applying the optimized depth residual network to the time sequence data set to generate a depth image; s5, dynamically adjusting an output layer of the deep residual network to adapt to environment change, and optimizing a generation strategy; s6, comparing with reference data, and optimizing deep residual network parameters by using error feedback; s7, performing fusion and post-processing on the generated depth information to obtain a final depth image; and S8, fusing the depth image and the sensor data, and carrying out environment modeling and real-time decision making. According to the method, the 3D camera, the depth residual network and the lion group optimization algorithm are fused, and high-precision and real-time depth information generation in a complex environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of 3D perception technology, and in particular to a method for real-time generation of depth information based on an integrated 3D perception camera module. Background Art

[0002] Over the past few decades, with the rapid advancement of science and technology, particularly in areas such as autonomous driving, robotic navigation, augmented reality, and virtual reality, three-dimensional perception has gradually become a key technology. 3D perception technology uses sensors such as cameras, lidar, and radar to acquire and process three-dimensional data of the surrounding environment, generating depth images or point cloud data, thereby enabling the perception and understanding of spatial information. In the field of autonomous driving, depth information is crucial for tasks such as vehicle positioning, path planning, and obstacle avoidance. Similarly, in robotics, depth perception helps robots perceive their surroundings, perform positioning, and dynamically avoid obstacles. To meet these needs, various depth perception technologies have gradually developed.

[0003] Among these technologies, 3D perception based on camera modules has gradually become one of the most popular. As lightweight, low-cost, and easily integrated sensors, cameras hold enormous potential in many applications. Traditional 3D perception technologies typically utilize stereo vision, structured light, and lidar, employing various computational and sensor fusion methods to acquire three-dimensional information. Stereo vision uses two or more cameras to capture different perspectives of a scene, calculating depth using the parallax between the images. Structured light, on the other hand, projects a specific pattern of light beams and uses changes in reflection from an object's surface to estimate depth. LiDAR technology, on the other hand, transmits and receives laser beams to calculate the distance between an object and the sensor, generating three-dimensional data about the environment.

[0004] However, although these technologies have solved the depth perception problem to a certain extent, they still have some shortcomings. First, stereo vision technology is very sensitive to changes in lighting, especially in low light or strong light conditions, its depth generation effect is significantly reduced. The influence of lighting makes it impossible for 3D perception systems based on stereo vision to provide accurate depth information in some environments. In addition, stereo vision also requires high-precision image alignment and calibration, and the amount of calculation is large, the real-time performance is poor, and it is difficult to meet the needs of application scenarios with high real-time requirements. On the other hand, although structured light can provide more accurate depth information, it depends on the quality and angle of the projected light beam, and is prone to errors when the surface of the object has complex reflective characteristics. In addition, LiDAR has certain advantages in accuracy, but its equipment is usually bulky and expensive, and in high-density environments (such as urban streets or indoor scenes), the detection range and resolution of LiDAR will be limited.

[0005] With the development of deep learning and artificial intelligence technologies, deep neural networks (DNNs) have been gradually introduced into the field of 3D perception, showing great potential in image feature extraction and depth information generation using deep residual networks. By using deep learning algorithms combined with deep residual networks, the system can automatically learn deep features in images, greatly improving the accuracy and robustness of depth image generation. Deep residual networks can learn more complex spatial features through a multi-layer network structure, significantly improving the adaptability of traditional methods in complex environments.

[0006] However, although deep residual networks have achieved remarkable results in image processing, they still face some challenges in depth information generation. First, traditional deep residual networks rely on a large amount of training data when processing complex environments, which requires the network to efficiently learn spatial features in different scenarios. However, deep learning models usually require a large amount of labeled data and computing resources to train efficient models. In some resource-constrained application scenarios, how to efficiently and accurately generate depth information remains an urgent problem to be solved. Secondly, deep residual networks may face high latency and computing bottlenecks in practical applications, especially in real-time depth information generation tasks. Even optimized network models may still not be able to respond quickly in a changing environment, resulting in the generated depth information not being real-time enough, affecting the decision-making efficiency of the system.

[0007] In addition, existing technologies also have great limitations when dealing with complex environmental changes (such as lighting changes, object occlusion, dynamic scenes, etc.). The impact of lighting changes on depth information generation is very significant, especially in the case of insufficient lighting or strong lighting, existing 3D perception technology is difficult to provide accurate depth data. In addition, in a dynamic environment, factors such as occlusion, reflection, and movement of objects will also affect the accuracy of depth information, and existing technologies are usually weak in dealing with these situations. Especially with the occlusion problem, in traditional depth generation methods, the depth calculation of the occluded area will be affected, resulting in missing or inaccurate generated depth images.

[0008] Therefore, how to provide a real-time generation method of depth information based on an integrated 3D perception camera module is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0009] One purpose of the present invention is to propose a method for real-time generation of depth information based on an integrated 3D perception camera module. The present invention makes full use of integrated 3D perception technology, deep residual network and lion group optimization algorithm, and describes in detail how to generate high-precision depth images in real time by optimizing the deep residual network structure and dynamically adjusting hyperparameters. By introducing the lion group optimization algorithm, the global search capability and local optimization capability of the depth information generation process are further improved, so that the method can cope with problems such as lighting changes and object occlusion under complex environmental conditions. The method has the advantages of strong real-time performance, high depth image accuracy, strong system robustness and wide adaptability, and can be widely used in fields such as autonomous driving, robot navigation, and intelligent monitoring.

[0010] According to an embodiment of the present invention, a method for real-time generation of depth information based on an integrated 3D perception camera module includes the following steps:

[0011] S1. Collect multi-source time series data of the integrated 3D perception camera module, pre-process the multi-source time series data, and generate a time series data set;

[0012] S2. Build a deep residual network, which includes multiple residual modules. Use the deep residual network to extract features from the time series dataset and generate a deep feature representation for each frame of the image.

[0013] S3. Introducing the Lion Group Optimization algorithm to optimize the hyperparameters in the deep residual network and optimize the deep residual network's ability to generate deep information in complex dynamic environments;

[0014] S4. Apply the optimized deep residual network to the time series dataset, generate depth information for each frame of image, and generate a depth image;

[0015] S5. In real-time applications, dynamically adjust the output layer of the deep residual network to automatically adapt the output of depth information generation according to environmental changes, and adjust the depth information generation strategy under complex lighting and object occlusion conditions;

[0016] S6. Based on the comparison between the generated depth image and the reference data of the actual scene, the error feedback mechanism is used to fine-tune the parameters of the deep residual network to optimize the accuracy of depth information generation and real-time response capability;

[0017] S7, fusing and post-processing the generated depth information, combining multi-view data and sensor information, eliminating errors, and obtaining the final depth image;

[0018] S8. Use the fused depth image and other sensor data to perform environmental modeling and dynamic analysis to support real-time decision-making applications.

[0019] Optionally, the multi-source time series data specifically includes RGB images, depth images and other sensor data, which are used to provide comprehensive perception information of the environment.

[0020] Optionally, the preprocessing of multi-source time series data specifically includes denoising, normalization and data alignment, which is used to improve the accuracy, stability and real-time performance of depth information generation.

[0021] Optionally, the generated depth image is constructed based on the depth feature representation of each frame of image.

[0022] Optionally, the S2 specifically includes:

[0023] S21. Design a network architecture of a deep residual network, wherein the deep residual network includes multiple residual modules, each residual module consists of a convolutional layer, a batch normalization layer, and an activation function layer, and each layer of the residual module is added through a residual connection;

[0024] S22. Select an appropriate convolution kernel size and step size. The convolution kernel size is k×k and the step size is s. In each convolution layer, the convolution kernel is convolved with the input image to extract spatial features in the image.

[0025] S23, using adaptive dynamic residual connections, automatically adjusts the connection mode in the residual module according to the complexity of the image and the changes in depth information, and introduces a dynamic gating mechanism to decide whether to strengthen or weaken the weight of the residual connection according to the changes in the characteristics of each frame of the image:

[0026] g(x)=τ(Wx+b)·(1+α·cos(2πfx));

[0027] Where τ is the activation function, W is the weight matrix, b is the bias term, x is the input feature, α is the adjustment factor, f is the frequency factor, g(x) represents the output of the dynamic gating function, and cos(2πfx) represents the periodic adjustment term.

[0028] S24. Perform batch normalization on the output of each residual module. The batch normalization operation standardizes the input data to eliminate the distribution differences between different layers:

[0029]

[0030] Among them, μ is the mean, σ 2 is the variance, ∈ is a small constant, γ and β are scaling factors and offsets respectively, ω is the frequency factor, and sin(ωx) is the periodic change adjustment term. Represents the standardized data;

[0031] S25, performing a nonlinear transformation on the output of the residual module through an activation function, wherein the activation function is a ReLU function;

[0032] S26. By introducing a multi-scale feature fusion mechanism, depth information is extracted at different scales, and the multi-scale combination of convolutional layers and pooling layers is used to enhance the perception ability of the deep residual network for objects of different sizes;

[0033] S27. Input the preprocessed time series data set into the deep residual network, perform feature extraction on each frame of image, and generate a deep feature representation of each frame of image, wherein the deep feature representation includes spatial information and depth information of each pixel in the image.

[0034] Optionally, the dynamic gating mechanism generates periodic adjustments as the input changes, so as to enhance the adaptability of the deep residual network to complex dynamic scenes.

[0035] Optionally, the S3 specifically includes:

[0036] S31. Initialize the individual population in the lion group optimization algorithm, where each individual is represented as a vector consisting of hyperparameters of the deep residual network, and the initial position of each individual is represented as a hyperparameter vector;

[0037] S32. Assign a fitness value to each individual. The fitness value is calculated by the objective function. The objective function is calculated by calculating the training error of the deep residual network under given hyperparameters or the error between the generated depth image and the actual image:

[0038]

[0039] Where MSE(p) is the mean square error under the hyperparameter configuration p, Var(p) is the variance of the depth image generated, β1 and β2 are adjustment coefficients, ρ 2 is a constant that controls the influence of variance, T is the period, t is the current time step, f(p) is the fitness function value, is a periodic adjustment item;

[0040] S33. Assign roles to individuals. Three individual roles are set: Lion King, Follower, and Explorer. Followers with poor performance will be transformed into Explorers for global search, while Explorers with excellent performance will be transformed into Followers for local optimization.

[0041] S34. Initialize the individual positions of the lion group optimization algorithm population. The individual positions are expressed as hyperparameter vectors, and the position of each individual is initialized in the search space. The initial positions are randomly distributed, and the quality of each individual is evaluated by the fitness function. When initialized, the position of each individual is And initialized to random values:

[0042]

[0043] in, represents the initial position of the i-th individual in the 0th iteration, and n represents the total number of hyperparameters;

[0044] S35. A dynamic gating mechanism is introduced to decide whether to switch roles based on the fitness of each individual and environmental changes. Followers with poor fitness will become explorers for a wider search, while explorers with excellent performance can become followers for local optimization. The evaluation function E for role switching is:

[0045] E=α1·f(P)+α2·distance(P,P global )+α3·convergence(P);

[0046] Among them, f(P) is the fitness value of the individual, distance(P,P global ) is the individual P and the global optimal solution P global The distance between them, convergence (P) is the degree of convergence between the individual and the historical optimal solution, α1, α2, α3 are adjustment factors;

[0047] S36. Update the position of the individual according to the lion group's behavior simulation. The updated position of the individual is not only guided by the position of the global optimal solution, but also influenced by the historical optimal solutions of other individuals:

[0048]

[0049] in, is the position of the i-th individual in the t+1th iteration, is the position in the current iteration, P best is the individual's historical optimal position, r1 and r2 are random numbers, c1 and c2 are learning factors, and α is the global search intensity of the explorer. is a position randomly selected from the entire search space;

[0050] S37. Updates are made based on the role of each individual. Followers' update positions are mainly guided by the lion king, while explorers conduct global searches through more extensive exploration. After the roles are dynamically switched, the update method of the individual will change. Individuals with poor fitness will turn into explorers and conduct more searches according to environmental changes.

[0051] S38, individuals optimize hyperparameters through continuous iteration. After each iteration, the fitness function value f(p) of each individual is evaluated, the hyperparameter configuration is updated, and the role switching is determined. During the hyperparameter optimization process, the fitness function value helps determine the optimal hyperparameter configuration;

[0052] S39, training according to the optimized hyperparameter configuration of the deep residual network, and training the deep residual network using the optimized hyperparameters;

[0053] S310. Dynamically adjust the optimized hyperparameters in real-time applications, use the final optimized deep residual network to generate depth information, and adjust the hyperparameters in real time during environmental changes.

[0054] Optionally, the hyperparameters include the number of convolution layers, the convolution kernel size, and the learning rate, which are used to optimize the ability to generate depth information and the training process of the deep residual network.

[0055] Optionally, the S4 specifically includes:

[0056] S41, loading each frame of the time series data set into the deep residual network for processing, and passing each frame of the image as input data to the optimized deep residual network for depth information generation;

[0057] S42. Using the optimized deep residual network to extract features from each frame of the image. The deep residual network extracts the spatial features of the image through the convolution layer and the residual module, and converts the image into a deep feature representation;

[0058] S43, using the decoder module of the deep residual network to process the extracted depth feature representation, generate a corresponding depth image, map the features of each frame image to a depth space, and output corresponding depth information;

[0059] S44, comparing the generated depth image with the reference depth data, evaluating the precision and accuracy of the generated result, and confirming whether the quality of the depth information generated meets the requirements;

[0060] S45. Adjust the hyperparameters or structure of the deep residual network according to the evaluation result of the generated depth image to optimize the quality of the generated depth image;

[0061] S46. Repeat the above steps to continuously optimize the deep residual network and the depth image generation process. In each iteration, the deep residual network will be fine-tuned according to the feedback and evaluation results to generate a more accurate and stable depth image.

[0062] Optionally, the S5 specifically includes:

[0063] S51. Obtaining real-time data on changes in the current environment, particularly lighting conditions and object occlusion, through external sensors or an environmental monitoring system. The change data is used to determine whether the current depth information generation process is affected by environmental factors.

[0064] S52, analyzing the environmental change data, extracting feature information related to illumination changes and object occlusion, analyzing the brightness distribution and contrast of the current image, and evaluating changes in ambient illumination;

[0065] S53. Dynamically adjust the output layer of the deep residual network based on the analysis results of the environmental change data. When the lighting conditions are poor or the object is severely occluded, adjust the parameters of the decoder layer or convolution layer of the network to adapt to different environmental conditions.

[0066] S54. Dynamically change the strategy for generating depth information based on the analysis results. Under complex lighting conditions, increase the adaptability of the deep residual network to low light conditions. Under object occlusion conditions, enhance the deep residual network's ability to process occluded areas.

[0067] S55, verifying the adjusted depth information generation result, by comparing it with the reference data of the actual scene, evaluating the accuracy and stability of the adjusted depth image in a complex environment, if the generated depth image meets the predetermined accuracy requirement, then continue to use the current adjustment strategy, otherwise further adjust the output layer or strategy;

[0068] S56. After each environment change, continue to fine-tune the output layer and generation strategy of the deep residual network.

[0069] The beneficial effects of the present invention are:

[0070] This paper introduces the Lion Group Optimization (Lion Group Optimization) algorithm to further enhance the deep residual network's ability to generate depth information in complex dynamic environments. Compared with traditional optimization methods, the Lion Group Optimization (Lion Group Optimization) algorithm can effectively optimize the hyperparameter configuration of the deep residual network, enhancing the network's global search and local optimization capabilities. By simulating the natural behavior of lions and combining the exploration and development capabilities of individuals in the search space, the Lion Group Optimization algorithm optimizes the structure and learning rate of the deep residual network, enabling the network to better adapt to different environmental changes, especially in the case of unstable lighting and object occlusion, ensuring the accuracy and real-time performance of the depth image.

[0071] In this paper, the Lion Group Optimization algorithm is used to dynamically optimize hyperparameters in deep residual networks, such as the number of convolutional layers, kernel size, and learning rate, to ensure that the network can generate high-precision depth images in different environments. By introducing explorer and lion king roles, the Lion Group Optimization algorithm can flexibly adjust the search strategy. When lighting changes or object occlusion affect depth generation, the Lion Group Optimization algorithm can perform global search and local optimization to avoid falling into local optimal solutions, thereby improving the stability and robustness of depth image generation.

[0072] Especially in real-time applications such as autonomous driving and robotic vision, the introduction of the Lion Group Optimization algorithm enables the system to not only optimize the accuracy of depth information generation, but also improve response speed and ensure the real-time performance of depth images. Through this innovative optimization mechanism, the system can efficiently generate depth images in complex and dynamic environments and quickly adapt to challenges such as changing lighting and object occlusion, significantly improving the performance of the depth perception system.

[0073] In summary, the application of the lion group optimization algorithm in the present invention not only enhances the optimization process of the deep residual network, but also improves the adaptability, accuracy and real-time performance of the system, overcomes the limitations of traditional depth perception technology in dynamic and complex environments, and has significant technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0075] Figure 1 This is a flow chart of the method for real-time generation of depth information based on an integrated 3D perception camera module proposed in the present invention;

[0076] Figure 2 This is a schematic diagram of the structure of the deep residual network based on the real-time generation method of depth information of the integrated 3D perception camera module proposed in the present invention;

[0077] Figure 3 This is a diagram showing the working principle of the lion group optimization algorithm proposed in the present invention based on the real-time generation method of depth information of the integrated 3D perception camera module. DETAILED DESCRIPTION

[0078] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0079] refer to Figure 1 、 Figure 2 and Figure 3 , a real-time depth information generation method based on an integrated 3D perception camera module includes the following steps:

[0080] S1. Collect multi-source time series data of the integrated 3D perception camera module, pre-process the multi-source time series data, and generate a time series data set;

[0081] S2. Build a deep residual network, which includes multiple residual modules. Use the deep residual network to extract features from the time series dataset and generate a deep feature representation for each frame of the image.

[0082] S3. Introducing the Lion Group Optimization algorithm to optimize the hyperparameters in the deep residual network and optimize the deep residual network's ability to generate deep information in complex dynamic environments;

[0083] S4. Apply the optimized deep residual network to the time series dataset, generate depth information for each frame of image, and generate a depth image;

[0084] S5. In real-time applications, dynamically adjust the output layer of the deep residual network to automatically adapt the output of depth information generation according to environmental changes, and adjust the depth information generation strategy under complex lighting and object occlusion conditions;

[0085] S6. Based on the comparison between the generated depth image and the reference data of the actual scene, the error feedback mechanism is used to fine-tune the parameters of the deep residual network to optimize the accuracy of depth information generation and real-time response capability;

[0086] S7, fusing and post-processing the generated depth information, combining multi-view data and sensor information, eliminating errors, and obtaining the final depth image;

[0087] S8. Use the fused depth image and other sensor data to perform environmental modeling and dynamic analysis to support real-time decision-making applications.

[0088] The present invention provides an accurate and efficient method for real-time generation of depth information by combining an integrated 3D perception camera module, a deep residual network and a lion group optimization algorithm. First, multi-source time series data and a deep residual network are used to extract deep features from images, effectively improving the accuracy of depth image generation. The introduction of the lion group optimization algorithm to optimize hyperparameters enhances the adaptability of the network in complex dynamic environments, enabling it to maintain good performance and accuracy under the influence of uncertain factors such as lighting changes and object occlusion. By adjusting the output layer of the deep residual network in real time, the system can dynamically adapt to environmental changes, further improving the accuracy and stability of depth image generation.

[0089] In addition, the error feedback mechanism of the present invention can optimize the parameters of the deep residual network in real time based on the comparison between the generated depth image and the actual scene data, thereby improving the accuracy and response speed of depth information generation. In the post-processing process, multi-view data and sensor information are combined to eliminate errors, ensuring that the generated depth image has high accuracy, which is suitable for applications such as autonomous driving and robot navigation that require high real-time and accuracy. Ultimately, by fusing the depth image with other sensor data, real-time decision analysis and environmental modeling are supported, which improves the overall performance and intelligence level of the system and has broad application prospects.

[0090] In this embodiment, the multi-source time series data specifically includes RGB images, depth images and other sensor data, which are used to provide comprehensive perception information of the environment.

[0091] In this embodiment, the preprocessing of multi-source time series data specifically includes denoising, normalization and data alignment, which is used to improve the accuracy, stability and real-time performance of depth information generation.

[0092] In this embodiment, the generated depth image is constructed based on the depth feature representation of each frame of image.

[0093] In this embodiment, S2 specifically includes:

[0094] S21. Design a network architecture of a deep residual network, wherein the deep residual network includes multiple residual modules, each residual module consists of a convolutional layer, a batch normalization layer, and an activation function layer, and each layer of the residual module is added through a residual connection;

[0095] S22. Select an appropriate convolution kernel size and step size. The convolution kernel size is k×k and the step size is s. In each convolution layer, the convolution kernel is convolved with the input image to extract spatial features in the image.

[0096] S23, using adaptive dynamic residual connections, automatically adjusts the connection mode in the residual module according to the complexity of the image and the changes in depth information, and introduces a dynamic gating mechanism to decide whether to strengthen or weaken the weight of the residual connection according to the changes in the characteristics of each frame of the image:

[0097] g(x)=τ(Wx+b)·(1+α·cos(2πfx));

[0098] Where τ is the activation function, W is the weight matrix, b is the bias term, x is the input feature, α is the adjustment factor, f is the frequency factor, g(x) represents the output of the dynamic gating function, and cos(2πfx) represents the periodic adjustment term.

[0099] S24. Perform batch normalization on the output of each residual module. The batch normalization operation standardizes the input data to eliminate the distribution differences between different layers:

[0100]

[0101] Among them, μ is the mean, σ 2 is the variance, ∈ is a small constant, γ and β are scaling factors and offsets respectively, ω is the frequency factor, and sin(ωx) is the periodic change adjustment term. Represents the standardized data;

[0102] S25, performing a nonlinear transformation on the output of the residual module through an activation function, wherein the activation function is a ReLU function;

[0103] S26. By introducing a multi-scale feature fusion mechanism, depth information is extracted at different scales, and the multi-scale combination of convolutional layers and pooling layers is used to enhance the perception ability of the deep residual network for objects of different sizes;

[0104] S27. Input the preprocessed time series data set into the deep residual network, perform feature extraction on each frame of image, and generate a deep feature representation of each frame of image, wherein the deep feature representation includes spatial information and depth information of each pixel in the image.

[0105] This paper proposes an image feature extraction method based on a deep residual network. By designing an innovative network architecture and optimizing residual modules, the accuracy and efficiency of depth information generation are significantly improved. First, the deep residual network extracts image features through multiple residual modules. Each residual module effectively avoids the vanishing gradient problem in deep networks through residual connections, enhancing the network's learning ability when processing complex images. The network architecture is designed so that each layer is added through residual connections, which helps retain more original image features and enables effective information transfer in deeper networks.

[0106] By selecting the appropriate convolution kernel size and step size, combined with an adaptive dynamic residual connection mechanism, the present invention can automatically adjust the network's connectivity based on the complexity and depth of each frame, enhancing the network's adaptability to varying environmental conditions. The dynamic gating mechanism automatically adjusts the weights of the residual connections based on changes in image features, enabling the network to process information more flexibly in complex scenes and improving the accuracy of image feature extraction.

[0107] Furthermore, batch normalization helps eliminate distribution differences between layers, accelerating network training. By introducing a multi-scale feature fusion mechanism, the deep residual network can extract depth information at different scales, thereby improving its ability to perceive objects of varying sizes. These optimizations enable the present invention to achieve high precision, robustness, and efficiency when handling depth information generation tasks in dynamically changing environments, enabling its widespread application in real-time depth perception systems such as autonomous driving and robotic vision.

[0108] In this embodiment, the dynamic gating mechanism generates periodic adjustments as the input changes, which is used to enhance the adaptability of the deep residual network to complex dynamic scenes.

[0109] In this embodiment, S3 specifically includes:

[0110] S31. Initialize the individual population in the lion group optimization algorithm, where each individual is represented as a vector consisting of hyperparameters of the deep residual network, and the initial position of each individual is represented as a hyperparameter vector;

[0111] S32. Assign a fitness value to each individual. The fitness value is calculated by the objective function. The objective function is calculated by calculating the training error of the deep residual network under given hyperparameters or the error between the generated depth image and the actual image:

[0112]

[0113] Where MSE(p) is the mean square error under the hyperparameter configuration p, Var(p) is the variance of the depth image generated, β1 and β2 are adjustment coefficients, ρ 2 is a constant that controls the influence of variance, T is the period, t is the current time step, f(p) is the fitness function value, is a periodic adjustment item;

[0114] S33. Assign roles to individuals. Three individual roles are set: Lion King, Follower, and Explorer. Followers with poor performance will be transformed into Explorers for global search, while Explorers with excellent performance will be transformed into Followers for local optimization.

[0115] S34. Initialize the individual positions of the lion group optimization algorithm population. The individual positions are expressed as hyperparameter vectors, and the position of each individual is initialized in the search space. The initial positions are randomly distributed, and the quality of each individual is evaluated by the fitness function. When initialized, the position of each individual is And initialized to random values:

[0116]

[0117] in, represents the initial position of the i-th individual in the 0th iteration, and n represents the total number of hyperparameters;

[0118] S35. A dynamic gating mechanism is introduced to decide whether to switch roles based on the fitness of each individual and environmental changes. Followers with poor fitness will become explorers for a wider search, while explorers with excellent performance can become followers for local optimization. The evaluation function E for role switching is:

[0119] E=α1·f(P)+α2·distance(P,P global )+α3·convergence(P);

[0120] Among them, f(P) is the fitness value of the individual, distance(P,P global ) is the individual P and the global optimal solution Pglobal The distance between them, convergence (P) is the degree of convergence between the individual and the historical optimal solution, α1, α2, α3 are adjustment factors;

[0121] S36. Update the position of the individual according to the lion group's behavior simulation. The updated position of the individual is not only guided by the position of the global optimal solution, but also influenced by the historical optimal solutions of other individuals:

[0122]

[0123] in, is the position of the i-th individual in the t+1th iteration, is the position in the current iteration, P best is the individual's historical optimal position, r1 and r2 are random numbers, c1 and c2 are learning factors, and α is the global search intensity of the explorer. is a position randomly selected from the entire search space;

[0124] S37. Updates are made based on the role of each individual. Followers' update positions are mainly guided by the lion king, while explorers conduct global searches through more extensive exploration. After the roles are dynamically switched, the update method of the individual will change. Individuals with poor fitness will turn into explorers and conduct more searches according to environmental changes.

[0125] S38, individuals optimize hyperparameters through continuous iteration. After each iteration, the fitness function value f(p) of each individual is evaluated, the hyperparameter configuration is updated, and the role switching is determined. During the hyperparameter optimization process, the fitness function value helps determine the optimal hyperparameter configuration;

[0126] S39. Perform training according to the optimized hyperparameter configuration of the deep residual network, and train the deep residual network using the optimized hyperparameters;

[0127] S310. Dynamically adjust the optimized hyperparameters in real-time applications, use the final optimized deep residual network to generate depth information, and adjust the hyperparameters in real time during environmental changes.

[0128] This paper introduces the Lion Pack Optimization (Lion Pack Optimization) algorithm to effectively improve the hyperparameter optimization process of deep residual networks, particularly the ability to generate deep information in complex dynamic environments. The three roles in the Lion Pack Optimization algorithm—the Lion King, the Follower, and the Explorer—each have distinct roles, and dynamic role switching achieves a balance between global and local optimization.

[0129] The introduction of the Explorer role is one of the key innovations of this invention. Through independent global search, the Explorer effectively avoids the local optimal solution problem that may be encountered in traditional optimization methods. When the environment undergoes drastic changes, the Explorer can quickly conduct extensive exploration, helping the system find new and potential optimization paths. The Explorer's global search capability ensures that the deep residual network can find the optimal hyperparameter configuration in complex and unknown environments, especially when faced with challenges such as dynamic changes and object occlusion.

[0130] At the same time, the Lion, representing the global optimal solution, guides followers in local optimization, helping the network make fine adjustments near the existing solution to ensure optimization accuracy. Followers, in turn, rely on the Lion for local search, continuously adjusting hyperparameter configurations to optimize the accuracy of depth information generation. Explorers, Lions, and Followers dynamically switch and collaborate, forming a highly efficient optimization mechanism. When a follower's fitness declines, it becomes an Explorer, conducting an extensive global search to find new potential solutions. High-performing Explorers transition to Followers, focusing on local optimization.

[0131] Through this coordination and cooperation between roles, the present invention achieves efficient optimization of the hyperparameters of the deep residual network, which not only improves the accuracy of depth image generation, but also enhances the adaptability and robustness of the system in complex environments.

[0132] In this embodiment, the hyperparameters include the number of convolution layers, the convolution kernel size, and the learning rate, which are used to optimize the ability to generate depth information and the training process of the deep residual network.

[0133] In this embodiment, the S4 specifically includes:

[0134] S41, loading each frame of the time series data set into the deep residual network for processing, and passing each frame of the image as input data to the optimized deep residual network for depth information generation;

[0135] S42. Using the optimized deep residual network to extract features from each frame of the image. The deep residual network extracts the spatial features of the image through the convolution layer and the residual module, and converts the image into a deep feature representation;

[0136] S43, using the decoder module of the deep residual network to process the extracted depth feature representation, generate a corresponding depth image, map the features of each frame image to a depth space, and output corresponding depth information;

[0137] S44, comparing the generated depth image with the reference depth data, evaluating the precision and accuracy of the generated result, and confirming whether the quality of the depth information generated meets the requirements;

[0138] S45. Adjust the hyperparameters or structure of the deep residual network according to the evaluation result of the generated depth image to optimize the quality of the generated depth image;

[0139] S46. Repeat the above steps to continuously optimize the deep residual network and the depth image generation process. In each iteration, the deep residual network will be fine-tuned according to the feedback and evaluation results to generate a more accurate and stable depth image.

[0140] By applying an optimized deep residual network to a time-series dataset, the present invention achieves high-precision depth information generation, significantly improving the accuracy and stability of the deep image generation process. First, the deep residual network extracts features from each frame of the image through convolutional layers and residual modules, effectively converting the image's spatial features into deep feature representations. This process optimizes the network structure, enabling the network to process different image features in complex environments, especially in dynamically changing scenes, significantly enhancing the ability to extract deep features.

[0141] Next, the decoder module of the deep residual network further processes the extracted depth features and generates a precise depth image. This process maps the features of each frame in the image to a depth space, allowing the depth information to more accurately reflect the three-dimensional structure of the scene. Furthermore, by comparing it with reference depth data, the precision and accuracy of the generated depth image can be evaluated in real time, providing clear feedback for optimizing depth information generation.

[0142] Another advantage of the present invention is that by adjusting the hyperparameters or structure of the deep residual network based on the evaluation results, the quality of depth image generation can be continuously optimized. During each iteration, the deep residual network is fine-tuned based on the feedback, ensuring that the generated depth image is more accurate and stable. Through this continuous optimization mechanism, the system can adapt to changes in different environments and generate higher-quality depth images to meet the high-precision requirements of real-time depth perception applications such as autonomous driving and robotic vision.

[0143] In this embodiment, the S5 specifically includes:

[0144] S51. Obtaining real-time data on changes in the current environment, particularly lighting conditions and object occlusion, through external sensors or an environmental monitoring system. The change data is used to determine whether the current depth information generation process is affected by environmental factors.

[0145] S52, analyzing the environmental change data, extracting feature information related to illumination changes and object occlusion, analyzing the brightness distribution and contrast of the current image, and evaluating changes in ambient illumination;

[0146] S53. Dynamically adjust the output layer of the deep residual network based on the analysis results of the environmental change data. When the lighting conditions are poor or the object is severely occluded, adjust the parameters of the decoder layer or convolution layer of the network to adapt to different environmental conditions.

[0147] S54. Dynamically change the strategy for generating depth information based on the analysis results. Under complex lighting conditions, increase the adaptability of the deep residual network to low light conditions. Under object occlusion conditions, enhance the deep residual network's ability to process occluded areas.

[0148] S55, verifying the adjusted depth information generation result, by comparing it with the reference data of the actual scene, evaluating the accuracy and stability of the adjusted depth image in a complex environment, if the generated depth image meets the predetermined accuracy requirement, then continue to use the current adjustment strategy, otherwise further adjust the output layer or strategy;

[0149] S56. After each environment change, continue to fine-tune the output layer and generation strategy of the deep residual network.

[0150] The present invention significantly improves the adaptability and accuracy of depth information generation by acquiring environmental change data in real time and dynamically adjusting the output layer and generation strategy of the deep residual network based on the analysis results. When faced with complex environments such as lighting changes and object occlusion, the deep residual network can automatically adjust the network's decoder layer or convolution layer parameters based on real-time monitored environmental changes to ensure the stability and accuracy of depth image generation. Especially in low-light conditions or when objects are severely occluded, the present invention can enhance the network's ability to handle lighting changes and occluded areas, generating high-precision depth images.

[0151] By introducing a dynamic adjustment mechanism, the present invention enables the depth information generation process to adapt to changes in different environments, reducing the impact of lighting changes and occlusion issues. This flexible adjustment strategy not only improves the system's robustness in complex environments, but also reduces depth image errors caused by changing environmental factors.

[0152] Furthermore, the verification step of the present invention further ensures the effectiveness of the adjustment strategy by comparing it with reference data from real-world scenarios. After each adjustment, the accuracy and stability of the depth image are verified and optimized in real time, ensuring that the system consistently generates high-quality depth images in diverse environments. This continuous fine-tuning process improves the accuracy and responsiveness of depth image generation, giving the present invention significant advantages in real-time applications such as autonomous driving and robotic vision.

[0153] Example 1:

[0154] To verify the feasibility of this invention, it was applied to the forward environment perception system of a self-driving car. The primary technical challenge addressed was generating depth information in complex, dynamic environments, specifically addressing the accuracy of depth image generation under varying lighting conditions and occlusion. This system, combining an integrated 3D perception camera module with a deep residual network, dynamically adjusts the output layer and generation strategy, effectively improving the accuracy and robustness of depth information generation.

[0155] At our autonomous driving test site, we selected a typical urban road scene featuring multiple dynamic elements (such as pedestrians, other vehicles, and trees), and with changing lighting conditions over time. This complex, dynamic environment is plagued by interference factors such as lighting changes (such as entering a tunnel or experiencing direct sunlight) and obstructions on the road (such as pedestrians crossing the road or a large truck blocking the view ahead). These complex environmental factors can affect the depth perception capabilities of autonomous driving systems. Traditional depth perception systems, in particular, are prone to inaccurate or distorted depth images under these conditions, impacting the system's ability to make real-time decisions.

[0156] In this test scenario, we collected environmental data by installing an integrated 3D perception camera module and a matching deep residual network. The camera module continuously captures each frame of image data and transmits this image data to the deep residual network for processing. The deep residual network first extracts features from each frame of the image and then generates the corresponding depth image through an optimized decoder. During this process, the system monitors environmental changes in real time, especially lighting changes and object occlusions, and optimizes depth image generation by dynamically adjusting the output layer and generation strategy of the deep residual network. Specifically, when lighting conditions vary greatly, the system will enhance the feature extraction capability in low-light environments; when there are objects blocking the view, the system will enhance the depth processing capability of the occluded area to ensure the integrity of the depth information.

[0157] The deep residual network's hyperparameters are optimized using the Lion Pack Optimization algorithm. The system automatically adjusts hyperparameters such as the number of convolutional layers, kernel size, and learning rate based on the complexity of each frame. Over time, the deep residual network continuously adjusts its structure and parameters based on feedback, ensuring that the generated deep images remain highly accurate and stable.

[0158] In this embodiment, we conducted a series of tests in urban road scenarios from October 1, 2024 to October 7, 2024. The test sites were selected from three main roads in City A, including intersections with high traffic volume and a tunnel section.

[0159] Table 1 Comparison of depth image generation effects under different environmental conditions

[0160]

[0161]

[0162] In Table 1 above, we compare the depth image generation effects under different environmental conditions and analyze the performance of the traditional method and the method of the present invention under conditions of changing lighting and object occlusion. The specific analysis is as follows:

[0163] Under normal lighting conditions, the error of the traditional method is 3.5%, while the error of the proposed method is reduced to 1.2%. This shows that the proposed method can generate higher-precision depth images under standard lighting conditions, significantly improving the accuracy of depth information generation, and is particularly suitable for scenarios with high precision requirements such as autonomous driving.

[0164] In strong lighting conditions, the error of the traditional method is 7.8%, while the error of the proposed method is 2.3%. This shows that the proposed method exhibits greater adaptability and robustness under strong lighting conditions, effectively reducing the impact of lighting on depth information generation. Traditional methods have large errors under strong lighting conditions, while the proposed method can provide more accurate depth images by optimizing network parameters.

[0165] Under tunnel lighting conditions, the error of the traditional method is 6.5%, while the error of the proposed method is reduced to 2.0%. This difference further demonstrates the advantages of the proposed method in low-light environments. Low-light environments often affect the depth image generation of traditional methods. The proposed method effectively addresses low-light issues by dynamically adjusting the generation strategy, ensuring depth image quality.

[0166] When objects are occluded, traditional methods suffer from large errors. In the case of slight occlusion, the traditional method has an error of 9.4%, while the proposed method has an error of 3.1%. In the case of severe occlusion, the traditional method has an error of 15.7%, while the proposed method has an error of 6.2%. These data show that traditional methods perform poorly in occluded environments, while the proposed method significantly improves the accuracy of depth information generation in occluded environments by optimizing the deep residual network and adjusting hyperparameters.

[0167] In terms of processing time, the processing time of the present invention remains stable (about 80ms to 125ms) under various test environments, and compared with traditional methods, it has higher real-time response capabilities, especially in complex environments, with lower processing time delays.

[0168] Overall, through the optimization and dynamic adjustment strategy of the deep residual network, the present invention can generate more accurate and stable depth images under different lighting changes and object occlusion environments. It has high adaptability and robustness, and is particularly suitable for application scenarios that require real-time depth perception, such as autonomous driving and robot navigation.

[0169] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for real-time generation of depth information based on an integrated 3D perception camera module, characterized in that: The steps include: S1. Collect multi-source time series data of the integrated 3D perception camera module, pre-process the multi-source time series data, and generate a time series data set; S2. Build a deep residual network, which includes multiple residual modules. Use the deep residual network to extract features from the time series dataset and generate a deep feature representation for each frame of the image. S3. Introducing the Lion Group Optimization algorithm to optimize the hyperparameters in the deep residual network and optimize the deep residual network's ability to generate deep information in complex dynamic environments; S4. Apply the optimized deep residual network to the time series dataset, generate depth information for each frame of image, and generate a depth image; S5. In real-time applications, dynamically adjust the output layer of the deep residual network to automatically adapt the output of depth information generation according to environmental changes, and adjust the depth information generation strategy under complex lighting and object occlusion conditions; S6. Based on the comparison between the generated depth image and the reference data of the actual scene, the error feedback mechanism is used to fine-tune the parameters of the deep residual network to optimize the accuracy of depth information generation and real-time response capability; S7, fusing and post-processing the generated depth information, combining multi-view data and sensor information, eliminating errors, and obtaining the final depth image; S8. Use the fused depth image and other sensor data to perform environmental modeling and dynamic analysis to support real-time decision-making applications.

2. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The multi-source time series data specifically includes RGB images, depth images and other sensor data, which are used to provide comprehensive perception information of the environment.

3. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The preprocessing of multi-source time series data specifically includes denoising, normalization and data alignment, which is used to improve the accuracy, stability and real-time performance of depth information generation.

4. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The generated depth image is constructed based on the depth feature representation of each frame of image.

5. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The S2 specifically includes: S21. Design a network architecture of a deep residual network, wherein the deep residual network includes multiple residual modules, each residual module consists of a convolutional layer, a batch normalization layer, and an activation function layer, and each layer of the residual module is added through a residual connection; S22. Select an appropriate convolution kernel size and step size. The convolution kernel size is k×k and the step size is s. In each convolution layer, the convolution kernel is convolved with the input image to extract spatial features in the image. S23, using adaptive dynamic residual connections, automatically adjusts the connection mode in the residual module according to the complexity of the image and the changes in depth information, and introduces a dynamic gating mechanism to decide whether to strengthen or weaken the weight of the residual connection according to the changes in the characteristics of each frame of the image: g(x)=τ(Wx+b)·(1+α·cos(2πfx)); Where τ is the activation function, W is the weight matrix, b is the bias term, x is the input feature, α is the adjustment factor, f is the frequency factor, g(x) represents the output of the dynamic gating function, and cos(2πfx) represents the periodic adjustment term. S24. Perform batch normalization on the output of each residual module. The batch normalization operation standardizes the input data to eliminate the distribution differences between different layers: Among them, μ is the mean, σ 2 is the variance, ∈ is a small constant, γ and β are scaling factors and offsets respectively, ω is the frequency factor, and sin(ωx) is the periodic change adjustment term. Represents the standardized data; S25, performing a nonlinear transformation on the output of the residual module through an activation function, wherein the activation function is a ReLU function; S26. By introducing a multi-scale feature fusion mechanism, depth information is extracted at different scales, and the multi-scale combination of convolutional layers and pooling layers is used to enhance the perception ability of the deep residual network for objects of different sizes; S27. Input the preprocessed time series data set into the deep residual network, perform feature extraction on each frame of image, and generate a deep feature representation of each frame of image, wherein the deep feature representation includes spatial information and depth information of each pixel in the image.

6. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 5, characterized in that: The dynamic gating mechanism produces periodic adjustments as the input changes, which is used to enhance the adaptability of the deep residual network to complex dynamic scenes.

7. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The S3 specifically includes: S31. Initialize the individual population in the lion group optimization algorithm, where each individual is represented as a vector consisting of hyperparameters of the deep residual network, and the initial position of each individual is represented as a hyperparameter vector; S32. Assign a fitness value to each individual. The fitness value is calculated by the objective function. The objective function is calculated by calculating the training error of the deep residual network under given hyperparameters or the error between the generated depth image and the actual image: Where MSE(p) is the mean square error under the hyperparameter configuration p, Var(p) is the variance of the depth image generated, β1 and β2 are adjustment coefficients, ρ 2 is a constant that controls the influence of variance, T is the period, t is the current time step, f(p) is the fitness function value, is a periodic adjustment item; S33. Assign roles to individuals. Three individual roles are set: Lion King, Follower, and Explorer. Followers with poor performance will be transformed into Explorers for global search, while Explorers with excellent performance will be transformed into Followers for local optimization. S34. Initialize the individual positions of the lion group optimization algorithm population. The individual positions are expressed as hyperparameter vectors, and the position of each individual is initialized in the search space. The initial positions are randomly distributed, and the quality of each individual is evaluated by the fitness function. When initialized, the position of each individual is And initialized to random values: in, represents the initial position of the i-th individual in the 0th iteration, and n represents the total number of hyperparameters; S35. A dynamic gating mechanism is introduced to decide whether to switch roles based on the fitness of each individual and environmental changes. Followers with poor fitness will become explorers for a wider search, while explorers with excellent performance can become followers for local optimization. The evaluation function E for role switching is: E=α1·f(P)+α2·distance(P,P global )+α3·convergence(P); Among them, f(P) is the fitness value of the individual, distance(P,P global ) is the individual P and the global optimal solution P global The distance between them, convergence (P) is the degree of convergence between the individual and the historical optimal solution, α1, α2, α3 are adjustment factors; S36. Update the position of the individual according to the lion group's behavior simulation. The updated position of the individual is not only guided by the position of the global optimal solution, but also influenced by the historical optimal solutions of other individuals: in, is the position of the i-th individual in the t+1th iteration, is the position in the current iteration, P best is the individual's historical optimal position, r1 and r2 are random numbers, c1 and c2 are learning factors, and α is the global search intensity of the control explorer. is a position randomly selected from the entire search space; S37. Updates are made based on the role of each individual. Followers' update positions are mainly guided by the lion king, while explorers conduct global searches through more extensive exploration. After the roles are dynamically switched, the update method of the individual will change. Individuals with poor fitness will turn into explorers and conduct more searches according to environmental changes. S38, individuals optimize hyperparameters through continuous iteration. After each iteration, the fitness function value f(p) of each individual is evaluated, the hyperparameter configuration is updated, and the role switching is determined. During the hyperparameter optimization process, the fitness function value helps determine the optimal hyperparameter configuration; S39, training according to the optimized hyperparameter configuration of the deep residual network, and training the deep residual network using the optimized hyperparameters; S310. Dynamically adjust the optimized hyperparameters in real-time applications, use the final optimized deep residual network to generate depth information, and adjust the hyperparameters in real time during environmental changes.

8. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 7, characterized in that: The hyperparameters include the number of convolutional layers, the convolution kernel size, and the learning rate, which are used to optimize the ability to generate depth information and the training process of the deep residual network.

9. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The S4 specifically includes: S41, loading each frame of the time series data set into the deep residual network for processing, and passing each frame of the image as input data to the optimized deep residual network for depth information generation; S42. Using the optimized deep residual network to extract features from each frame of the image. The deep residual network extracts the spatial features of the image through the convolution layer and the residual module, and converts the image into a deep feature representation; S43, using the decoder module of the deep residual network to process the extracted depth feature representation, generate a corresponding depth image, map the features of each frame image to a depth space, and output corresponding depth information; S44, comparing the generated depth image with the reference depth data, evaluating the precision and accuracy of the generated result, and confirming whether the quality of the depth information generated meets the requirements; S45. Adjust the hyperparameters or structure of the deep residual network according to the evaluation result of the generated depth image to optimize the quality of the generated depth image; S46. Repeat the above steps to continuously optimize the deep residual network and the depth image generation process. In each iteration, the deep residual network will be fine-tuned according to the feedback and evaluation results to generate a more accurate and stable depth image.

10. The method for real-time generation of depth information based on an integrated 3D perception camera module according to claim 1, characterized in that: The S5 specifically includes: S51. Obtaining real-time change data of the current environment, particularly lighting conditions and object occlusion, through an external sensor or environmental monitoring system. The change data is used to determine whether the current depth information generation process is affected by environmental factors. S52, analyzing the environmental change data, extracting feature information related to illumination changes and object occlusion, analyzing the brightness distribution and contrast of the current image, and evaluating changes in ambient illumination; S53. Dynamically adjust the output layer of the deep residual network based on the analysis results of the environmental change data. When the lighting conditions are poor or the object is severely occluded, adjust the parameters of the decoder layer or convolution layer of the network to adapt to different environmental conditions. S54. Dynamically change the strategy for generating depth information based on the analysis results. Under complex lighting conditions, increase the adaptability of the deep residual network to low light conditions. Under object occlusion conditions, enhance the deep residual network's ability to process occluded areas. S55, verifying the adjusted depth information generation result, by comparing it with the reference data of the actual scene, evaluating the accuracy and stability of the adjusted depth image in a complex environment, if the generated depth image meets the predetermined accuracy requirement, then continue to use the current adjustment strategy, otherwise further adjust the output layer or strategy; S56. After each environment change, continue to fine-tune the output layer and generation strategy of the deep residual network.

Images