A synthetic aperture imaging autofocusing method based on trajectory prediction

Abstract

An autofocusing method for synthetic aperture imaging based on trajectory prediction includes the following steps: S100: A multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes a target pedestrian and acquires pedestrian trajectory data; S200: A neural network is constructed to obtain hybrid features, and the pedestrian trajectory is predicted based on the hybrid features, and the predicted pedestrian trajectory is corrected; S300: The predicted pedestrian trajectory is converted into a point in the synthetic depth sequence of the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system and synthesized onto a specified plane; S400: The relevant data is stored in an external storage device. This method can accurately predict pedestrian trajectories in occluded scenes and guide the autofocusing of synthetic aperture imaging based on the predicted pedestrian trajectory, thereby clearly observing frequently occluded pedestrians, which plays an important role in fields such as assisted driving and intelligent monitoring.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of computer vision and image processing, and particularly relates to a synthetic aperture imaging automatic focusing method based on trajectory prediction. BACKGROUND

[0002] The occlusion, illumination transformation, complex background, and mutual interference between observation targets in the occlusion scene environment are important contents restricting the development of computer vision technology. How to perform high-precision trajectory prediction in the occlusion environment is a key to the safe and efficient operation of current autonomous driving and mobile intelligent agents. Observing frequently occluded pedestrians in the occlusion scene, extracting features from the observed information and accurately predicting the pedestrian trajectory, and further observing the pedestrian using the predicted pedestrian trajectory to avoid collision with the pedestrian in the scene are the keys to the problem.

[0003] Early pedestrian trajectory prediction mainly focuses on modeling interactive behavior based on handcrafted features, such as social force models, physical mechanics models, and interactive Gaussian processes. Statistical model-based methods, such as hidden Markov models, discrete choice frameworks, and Gaussian processes, also show good results on related datasets. With the development of deep learning, neural network models are used to adaptively extract pedestrian trajectory features and predict pedestrian trajectories. The LSTM-based trajectory prediction method regards the pedestrian trajectory as a set of time series, and the social pooling layer trains the model by updating the LSTM hidden state variable with the self-state and social interaction information.

[0004] For a long time, pedestrian trajectory research has focused on the bird's eye view, and the first-person perspective can obtain key information such as pedestrian scale, pedestrian pose, and pedestrian motion. However, in complex scenes, it is difficult to observe pedestrians due to frequent and complex occlusions. Synthetic aperture imaging technology can be used to observe occluded targets well. However, synthetic aperture imaging usually uses exhaustive focusing and manual screening of clear images, and the automatic focusing algorithm often focuses on the plane where the occlusion is located, which cannot meet the actual needs. SUMMARY

[0005] To solve the above problems, the present disclosure provides a synthetic aperture imaging automatic focusing method based on trajectory prediction, comprising the following steps:

[0006] S100: Multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes a target pedestrian and obtains pedestrian trajectory data;

[0007] S200: Constructing a neural network, obtaining mixed features, predicting a pedestrian trajectory based on the mixed features, and correcting the predicted pedestrian trajectory;

[0008] S300: convert a point in the synthesized depth sequence by the predicted pedestrian trajectory into a multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experiment system and synthesize to a specified plane;

[0009] S400: store the relevant data into an external storage device.

[0010] Through the above technical solution, the pedestrian trajectory is accurately predicted in the occlusion scene where there is frequent mutual occlusion between pedestrians and the pedestrians are occluded by obstacles in the environment, and the synthetic aperture imaging automatic focusing is guided according to the predicted pedestrian trajectory, so that the frequently occluded pedestrians are clearly observed. It plays an important role in the fields of auxiliary driving, intelligent monitoring and the like. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of a trajectory prediction-based synthetic aperture imaging automatic focusing method provided in one embodiment of the present disclosure;

[0012] Figure 2 is a neural network structure diagram in one embodiment of the present disclosure;

[0013] Figure 3 is a feature extraction module structure diagram in one embodiment of the present disclosure;

[0014] Figure 4 is an intention attention unit structure diagram in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In order for those skilled in the art to understand the technical solutions disclosed in the present disclosure, the following will be combined with the embodiments and related drawings Figures 1 to 4 The technical solutions of various embodiments are described, and the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The terms "first", "second" and the like used in the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, "include" and "have" and any variations thereof are intended to cover and not exclusive inclusion. For example, a process, or method, or system, or product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, system, product or device.

[0016] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all

[0017] In one embodiment, as shown in Figure 1 A method for providing a trajectory prediction-based synthetic aperture imaging autofocus is disclosed, comprising the following steps:

[0018] S100: A multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes pedestrians and acquires pedestrian trajectory data;

[0019] S200: A neural network is constructed, mixed features are acquired, pedestrian trajectories are predicted based on the mixed features, and the predicted pedestrian trajectories are corrected;

[0020] S300: The predicted pedestrian trajectories are converted into a point in the synthetic depth sequence of the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system and synthesized into a specified plane;

[0021] S400: Relevant data is stored in an external storage device.

[0022] For this embodiment, the method consists of data acquisition and acquisition, data labeling, coordinate conversion, establishment of a pedestrian trajectory prediction model, training of a neural network, and synthetic aperture imaging autofocus steps. Pedestrian trajectories are acquired based on a first-person perspective through a multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system, pedestrian trajectory data is acquired based on synthetic aperture imaging technology, and social relationship information between pedestrians and scene context semantic information are acquired based on a first-person perspective through a multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system. Modeling is performed based on the acquired information, mixed features are acquired, pedestrian trajectories are predicted based on the mixed features, and the predicted pedestrian trajectories are corrected in combination with pedestrian intentions. The observed pedestrian trajectory coordinates are mapped to a synthetic aperture imaging focusing depth sequence, and conversion is performed between the image acquisition and imaging focusing depth sequence of the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system and the real three-dimensional coordinates.

[0023] In another embodiment, step S100 further comprises:

[0024] S101: A multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes pedestrians and acquires pedestrian trajectory data;

[0025] S102: convert the collected image information into real-world coordinates of the pedestrian;

[0026] S103: extract the pedestrian intention, social relationship between pedestrians and scene context semantic information in the observation information by means of the neural network based on attention mechanism, as the key factors for pedestrian trajectory prediction.

[0027] For this embodiment, in a blocking scene S containing N agents, at time t, the spatial coordinates of the agents are represented as x∈S. By SAI observing agent A i the historical trajectory of agent A in a period of time i the trajectory of agent A in the next period of time i and automatically focus SAI on the depth of agent A j for observation at t∈[τ+1: T] to eliminate the blocking of other agents A i or other objects in the scene to A

[0028] Five scenes from the public data sets ETH and UCY are used as data sets for cross-validation, and the trajectory of pedestrian A i includes two parts: historical trajectory and real trajectory:

[0029]

[0030]

[0031] wherein, A i refers to the trajectory composed of the coordinate points of agent A at different times t, t∈[0, τ], A i refers to the trajectory composed of the coordinate points of agent A at time t, t∈[τ+1, T].

[0032] Pedestrian trajectories are selected from the public data set Apollo and divided into training set and test set in proportion, which are used to train the model to predict the pedestrian trajectory in the first-person perspective.

[0033] A blocking scene containing multiple pedestrians and blockers is constructed, and scene image information is collected by a high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system of a multi-camera array.

[0034] The multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system restores the relative positions between cameras through a planar and parallax method. Utilizing the relationship between the planar and parallax, the image acquired by the array can be refocused. The device achieves perspective functionality by simulating the lens imaging process of a large aperture and shallow depth of field.

[0035] If the plane II containing the object does not coincide with the camera's reference plane π, there will be a parallax Δρ between images captured by different cameras. i Camera C i With reference camera C ref The parallax Δρ between i This can be expressed as the relative position Δc between the two cameras. i With relative depth The product of:

[0036]

[0037] Images collected from different perspectives I i After affine transformation W i Projected onto the reference image:

[0038]

[0039] Here, E is the identity matrix, O is the zero matrix, and H... i Image I i Image I from the perspective of the reference camera ref The homography matrix. By overlaying the projected images pixel by pixel and averaging, the synthetic aperture image can be obtained:

[0040]

[0041] Here, I sAI Synthetic aperture image, where n is the number of cameras in the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system, and I i Refers to the RGB image captured by the i-th camera.

[0042] The images acquired by the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system are synthesized according to depth. By mapping the synthesized depth sequence to the actual depth of the target, the true depth information of the observed pedestrian can be determined. Specifically, assuming the depth of the observed target is y... ρ There is a linear mapping between the SAI disparity Δρ and the SAI disparity, and approximate solutions for parameters m1 and m2 can be obtained using the least squares method:

[0043] m1*y ρ +m2=△ρ (6)

[0044] Where m1 and m2 are the depths y that describe the observed target.ρ The parameters relating to the linear mapping relationship between the SAI disparity Δρ and the SAI.

[0045] Thus, from the data collected by the calibrated camera, it is only necessary to know the parallax Δρ between the images. i This allows the calculation of the depth of the observed target. Once the depth information is acquired, a scale r can be established during the calibration process, relating the number of pixels occupied by the object to its actual size. It is also assumed that a linear mapping exists between the depth of the observed target and the scale r:

[0046] m′1*y ρ +m′2=r (7)

[0047] Where m′1 and m′2 are the depths y that describe the observed target. ρ There is a linear mapping relationship between the scale r and the actual size of the object. m′1 and m′2 can be calculated using the least squares method. Thus, after obtaining the pedestrian's disparity information and pixel coordinates, this information can be converted into the pedestrian's world coordinates.

[0048] Treating pedestrian trajectories as time series, combined with pedestrian displacement Instead of directly using pedestrian spatial coordinates, predictions are made. During feature encoding, the model employs a self-attention mechanism to extract pedestrian behavior (a), scene information (e), social interactions between pedestrians (s), and the intrinsic relationships between these key elements from historical trajectories. Here, a, e, and s are modeled using a multilayer perceptron (f) with the same structure.

[0049]

[0050] here, The information refers to Ai's own behavioral state and the scene information at time t, and xj (j≠i) refers to the world coordinates of other intelligent agents in the scene.

[0051] The information in step S102 is acquired by a multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system; the image is an RGB image, which includes visual information such as scene information, occlusion information, and pedestrian information; pedestrian trajectory data is obtained by extracting pedestrian information. Specifically, the observed pedestrian trajectory data is obtained by converting the pedestrian pixel coordinates and their synthetic depth sequence positions into coordinate points in the world coordinate system. The "observation information" in step S103 refers to the pedestrian trajectory composed of pedestrian coordinates at different times, converted from the observed "image information".

[0052] In another embodiment, step S101 further includes:

[0053] Extract all pedestrian coordinates in the occlusion scene, observe the occluded pedestrians by synthetic aperture imaging technology and extract coordinate information, convert the extracted coordinate information into real-world coordinates, and obtain trajectory data of the pedestrians based on the extracted coordinate information.

[0054] In another embodiment, the neural network comprises a feature encoding module, a feature decoding module, and a destination unit, wherein the feature encoding module is used to extract and integrate information from observed historical trajectories; the feature decoding module is used to integrate feature information from the feature encoding module and the destination unit; the destination unit extracts final destination information of the pedestrians, and inputs the integrated final destination information into the feature decoding unit to correct the predicted pedestrian trajectory.

[0055] For this embodiment, as shown in Figure 2 , the neural network is composed of three parts: feature encoding, feature decoding, and destination unit.

[0056] In another embodiment, the feature encoding module comprises an intention attention unit, a time attention unit, and a hybrid attention unit, wherein the intention attention unit is used to extract features and combine the features of the pedestrian's own intention, the social information between pedestrians, and the contextual semantic information in the scene together, the time attention unit is used to extract the timing information in the pedestrian trajectory features, and the hybrid attention unit is used to fuse and reorganize the extracted features.

[0057] For this embodiment, as shown in Figure 3 , the feature encoding module comprises an intention attention unit, a time attention unit, and a hybrid attention unit. As shown in Figure 4 , the intention attention unit comprises an agent attention, a scene attention, and a social attention.

[0058] In another embodiment, step S200 further comprises:

[0059] S201: modeling the key factors of the pedestrian trajectory prediction by the feature encoding module, obtaining the probability distribution of the historical trajectory and converting it into a feature variable;

[0060] S202: decoding the feature variable into a multi-modal pedestrian coordinate point probability distribution step by step by the feature decoding module, and generating the future coordinates of the pedestrians according to the Gaussian mixture model;

[0061] S203: in the training phase, encoding the pedestrian's own intention and the social relationship between pedestrians by the destination unit, inputting into the feature decoding module, and correcting the weight in the feature decoding module according to the future coordinates of the pedestrians, so that the predicted pedestrian trajectory information is more consistent with the real situation.

[0062] For this embodiment, for a given historical trajectory x obs , x pred is obtained from the posterior distribution p(xpred x obs , ζ) is derived from the posterior distribution p(ζ|x obs ) given x obs . ζ allows the generation of x pred with multi-modal properties given x pred , modeling the uncertainty of the agent's own will. Here, x is generated indirectly through the agent's displacement predicted displacement sequence The probability distribution of the predicted displacement sequence

[0063]

[0064] The neural network consists of three parts: feature encoding, feature decoding, and goal unit. The feature encoding module has three units to extract and integrate information from the observed historical trajectory. Among them, the intention unit is used to integrate and extract the hidden intention of the agent, social relationship and scene context semantic information of the pedestrian in different states:

[0065]

[0066] Here, refers to the agent's own behavior state at time t, the scene information and the social relationship between pedestrians after the information processing of the attention unit. Q, K, and V refer to the query, key, and value in the self-attention mechanism, respectively, and dim K is the dimension of the key value.

[0067] The self-attention mechanism is mainly used in the intention unit to extract information, and multiple intention units have the same network structure but do not share weights. The output information is considered as a set of time series, and the extracted information is integrated and sent to the time attention unit according to the following formula:

[0068]

[0069] Here, refers to the output of the intention attention unit. The intention attention unit mainly models the observed pedestrian trajectory and fuses the obtained information to extract the possible intention of the pedestrian at different times.

[0070] The time attention unit adaptively integrates the intention of the agent A i at different times. Specifically, the time attention unit is used to integrate feature information from different time steps, assign different weights to these information, and integrate:

[0071]

[0072] The time attention unit integrates the outputs of the intention attention unit at different time steps t, t ∈ [0, τ] through an attention mechanism, denotes the output of the time attention unit.

[0073] The hybrid attention unit is used to integrate information of different dimensions and encode the features into sampling probabilities:

[0074]

[0075] Here, denotes the output of the hybrid attention unit. The intention attention unit, the time attention unit, and the hybrid attention unit are sequentially connected to form a feature encoding module of the neural network model. The output of the hybrid attention unit fuses the extracted features and is also regarded as the output of the feature encoding module.

[0076] At this time, the generation model probability can be expressed as:

[0077]

[0078] where is the hidden variable of the VAF, where i is the number of the agent Ai. The hidden variables at different time steps t are independent of each other and follow a Gaussian distribution.

[0079] In the integral, can be transformed into where κ will be optimized through a neural network. means that and the hidden variable can be sampled from the observed trajectory: Here λ is an optimizable parameter.

[0080] In order to make the prediction result closer to the true situation, during the training, the real trajectory is sampled, and similar to the sampling of the observed historical trajectory , the is also converted into a relative displacement and is processed using an attention-based method:

[0081]

[0082] Here, is obtained by having the same network structure as the model in the foregoing, for mining and integrating the hidden self-intention and interaction between different agents in the sample as the output of the destination unit, for extracting the destination information of the pedestrian in the training phase. During the training, the features output by the feature encoding module are sent to the feature decoding module.

[0083] Unlike the sampling process, in the decoding process, the gated recurrent unit based autoregressive model is used to generate the predicted trajectory step by step, and the current generated trajectory point is fed back to the model to generate the next trajectory point until the complete predicted trajectory is generated Specifically, after obtaining the hidden variable , it is sent to the decoder together with the extracted sample ζ. The decoder consists of a Gaussian mixture model and a gated recurrent unit. Each trajectory point is generated by the Gaussian mixture model according to the intrinsic relationship between the last node gated recurrent unit hidden variable and the output The input of the next gated recurrent unit node is generated and the input The gated recurrent unit is responsible for modeling the trajectory information of the input node and the social interaction, scene situation and other information contained in the last node hidden variable and updating the hidden variable to

[0084] In the training process, the distribution parameters κ, λ, η in the model will be optimized. First, p κ , p λ , q η are converted into multivariate Gaussian distribution through neural network:

[0085]

[0086] respectively represent that the distribution is subject to mean μ κ , μ λ , μ η and variance ∑ κ κ, ∑ λ λ, ∑ η η. p κ , p λ , q η are obtained through the feature decoder, the feature encoder and the goal attention unit respectively, and are refreshed in the feature decoder.

[0087] Similar to CVAF, in the training, the variational lower bound associated with the observed displacement is introduced and maximized, and in addition, the predicted displacement is restored to the trajectory coordinates The final loss function is defined as:

[0088] ​

[0089] where the first term is used to make the predicted trajectory closer to the true value . represents the KL divergence between the predicted trajectory and the true trajectory, encouraging the predicted trajectory to be close to the true value in distribution.

[0090] In another embodiment, the step S300 further comprises:

[0091] By the predicted trajectory information, the camera array is directly focused to the plane where the pedestrian is located, or is focused to the plane where the future coordinates of the pedestrian are located in advance, without the need for a traversal focus.

[0092] For this embodiment, after obtaining the predicted trajectory, the corresponding synthetic aperture imaging synthetic depth can be obtained through formulas (3)-(7), and the target pedestrian can be well observed by focusing to the corresponding depth. The conventional focusing method needs to manually select an appropriate plane from the synthetic depth sequence, and the present method can convert the predicted pedestrian coordinates into the depth that needs to be synthesized by mapping the world coordinate system to the synthetic depth sequence, thereby realizing automatic focusing.

[0093] In another embodiment, the relevant data in the step S400 include the observed pedestrian image, the extracted pedestrian trajectory data, the trained neural network weight, the predicted pedestrian trajectory data, and the image synthesized to the specified plane.

[0094] In another embodiment, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the trajectory prediction-based synthetic aperture imaging automatic focusing method according to any one of claims 1-8 when executing the computer program.

[0095] In another embodiment, a computer program product comprising instructions, which, when executed on a computer, cause the computer to perform the steps of the trajectory prediction-based synthetic aperture imaging automatic focusing method according to any one of claims 1-8.

[0096] Although the embodiments of the present application are described above with reference to the accompanying drawings, the present application is not limited to the specific embodiments and application fields described above, and the specific embodiments described above are merely illustrative and instructive, but are not restrictive. Those skilled in the art can make many forms under the guidance of the present specification and without departing from the scope protected by the claims of the present application, which are all included in the protection of the present application.

Claims

1. A trajectory prediction based synthetic aperture imaging autofocus method, comprising the following steps: S100: A multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes a pedestrian and acquires pedestrian trajectory data; S200: A neural network is constructed, mixed features are acquired, the pedestrian trajectory is predicted based on the mixed features, and the predicted pedestrian trajectory is corrected; S300: A point in the synthetic depth sequence of the multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system is converted from the predicted pedestrian trajectory and synthesized to a specified plane; S400: Relevant data is stored in an external storage device; Step S300 further comprises: The camera array is directly focused on the plane where the pedestrian is located through the predicted trajectory information, or is focused on the plane where the future coordinates of the pedestrian are located in advance, without the need for traversal focusing.

2. The method of claim 1, wherein step S100 further comprises: S101: A multi-camera array high-precision high-speed synchronous acquisition and synthetic aperture imaging experimental system observes a pedestrian and acquires pedestrian trajectory data; S102: The acquired image information is converted into real-world coordinates of the pedestrian; S103: A neural network based on an attention mechanism is used to extract the pedestrian intention, the social relationship between pedestrians, and the scene context semantic information in the observation information as key factors for pedestrian trajectory prediction.

3. The method of claim 2, wherein step S101 further comprises: All pedestrian coordinates in an occluded scene are extracted, an occluded pedestrian is observed through a synthetic aperture imaging technology, and coordinate information is extracted, which is converted into real-world coordinates, and the trajectory data of the pedestrian is acquired based on the extracted coordinate information.

4. The method of claim 1, the neural network comprising a feature encoding module, a feature decoding module, and a destination unit, wherein, The feature encoding module is used to extract and integrate information from the observed historical trajectory; The feature decoding module is used to integrate feature information from the feature encoding module and the destination unit; the destination unit extracts the final destination information of the pedestrian, and inputs the integrated final destination information into the feature decoding unit to correct the predicted pedestrian trajectory.

5. The method of claim 4, the feature encoding module comprising an intent attention unit, a temporal attention unit, and a hybrid attention unit, wherein, The intention attention unit is used to extract features and combine the pedestrian's own intention, the social information between pedestrians, and the scene context semantic information, the time attention unit is used to extract the timing information in the pedestrian trajectory features, and the mixed attention unit is used to fuse and reorganize the extracted features.

6. The method of claim 1, wherein step S200 further comprises: S201: The feature encoding module is used to model the key factors for pedestrian trajectory prediction, obtain the probability distribution of the historical trajectory, and convert it into a feature variable; S202: The feature decoding module is used to gradually decode the feature variable into a multi-modal pedestrian coordinate point probability distribution, and generate the future coordinates of the pedestrian according to a Gaussian mixture model; S203: In the training stage, the destination unit is used to encode the pedestrian's own intention and the social relationship between pedestrians, input into the feature decoding module, and correct the weights in the feature decoding module according to the future coordinates of the pedestrian, so that the predicted pedestrian trajectory information is more consistent with the actual situation. 7.The method of claim 1, wherein the related data in step S400 comprises observed pedestrian images, extracted pedestrian trajectory data, trained neural network weights, predicted pedestrian trajectory data, and images synthesized to a specified plane.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the trajectory prediction based synthetic aperture imaging autofocus method in any one of claims 1 to 7 when the computer program is executed. 9.A computer program product comprising instructions which, when executed on a computer, cause the computer to carry out the steps of the trajectory prediction based synthetic aperture imaging autofocus method of any one of claims 1 to 7.

Images