Vehicle function control method and device based on aerial track recognition

Abstract

The invention discloses a vehicle function control method and device based on aerial track recognition. The method comprises the following steps: triggering to enter a working mode for recognizing a motion track of a vehicle control device; performing real-time tracking and positioning on the moving vehicle control device through vision and inertial navigation; a real-time tracking and positioning result of the vehicle control device is used as input of a pre-trained neural network model, and the model outputs a motion track analyzed by the vehicle control device; and matching a to-be-controlled function corresponding to the motion trail, and instructing the vehicle-mounted machine system to control the vehicle to execute the to-be-controlled function. According to the scheme, the flexibility, accuracy, safety and interestingness of vehicle function control are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle control, and particularly relates to a vehicle function control method and device based on air trajectory recognition. Background Art

[0002] Currently, there are mainly the following three types of interaction methods for vehicle function control, but each has technical defects: 1. Traditional physical buttons are limited by the fixed layout of the mechanical structure, and there are problems such as a single interaction dimension and poor function scalability. There is a contradiction between its function capacity and the development of vehicle intelligence. Specifically, the physical arrangement density of the button group is negatively correlated with the user's operation accuracy, and the operation difficulty increases in complex function scenarios, posing a potential threat to driving safety.

[0003] 2. The touch screen virtual button system breaks through the physical space limitation, but has the defect of lacking tactile feedback, resulting in the difficulty for users to perform blind operations in a non-gazing state, and there are also potential safety hazards.

[0004] 3. When controlling vehicle functions by voice, it faces the problem of recognition rate attenuation caused by signal-to-noise ratio degradation in complex acoustic scenarios. Especially in the scenarios of high-speed vehicle driving or multiple occupants, the reliability of voice recognition significantly decreases. Summary of the Invention

[0005] The present invention aims to ensure the flexibility, accuracy, safety, and interestingness of vehicle function control, and provides a vehicle function control method and device based on air trajectory recognition.

[0006] To achieve this purpose, the present invention adopts the following technical solutions: Provide a vehicle function control method based on air trajectory recognition, including the steps of: S1. Trigger and enter the working mode for recognizing the movement trajectory of the vehicle control device. S2. Real-time track and position the moving vehicle control device through vision plus inertial navigation. S3. Use the real-time track and position result of step S2 as the input of a pre-trained neural network model, and the model outputs the analyzed movement trajectory of the vehicle control device. S4. Match the quasi-control function corresponding to the movement trajectory, and instruct the vehicle-mounted system to control the vehicle to execute the quasi-control function.

[0007] Preferably, in step S1, the method for triggering and entering the working mode for recognizing the movement trajectory of the movable vehicle control device is: The vehicle control device generates a trigger command based on a user pressing a physical button provided on the vehicle control device, and sends the trigger command to the vehicle head unit system. The vehicle head unit system controls the vision recognition device and / or the inertial navigation device provided in the vehicle control device to enter a working mode for recognizing the motion trajectory of the vehicle control device, or the inertial navigation device directly enters a working mode for recognizing the motion trajectory of the vehicle control device based on the user pressing the physical button.

[0008] Preferably, step S2 specifically includes the steps of: S21, constructing a road point feature matrix , filtering out road points for characterizing the system state of the vehicle control device in the device body coordinate system in each frame of consecutive frames, where the road points are three-dimensional space points with a preset detected feature in the vehicle cockpit environment; S22, according to each of the road points filtered out in step S21, and by means of an inverse depth representation method, representing the system state of the vehicle control device in the device body coordinate system.

[0009] Preferably, in step S21, the method for filtering out road points from each frame of image specifically includes the steps of: A1, selecting a pixel point in each frame of consecutive frames , and defining the brightness of the pixel point as , then taking the pixel point as the center, and selecting pixel points on a circle with a radius of ; A2, judging whether the number of pixel points with brightness greater than or brightness less than among the pixel points selected in step A1 is greater than a preset number threshold, if so, extracting the pixel point as a candidate corner point and proceeding to step A3; if not, not taking the pixel point as the candidate corner point; A3, returning to step A1, cyclically traversing all pixel points in the same frame of image, and after completing the cyclic traversal, calculating the road point scores for all the candidate corner points extracted from the same frame of image; A4, selecting the top candidate corner points ranked from high to low score as the respective road points filtered out for this frame of image; Preferably, in step A1, is 3 pixels. It is 16 pixels and the 16 pixels are evenly arranged in the circumferential direction; The quantity threshold in step A2 is 12; In step A3, the method for calculating the road sign point score of the candidate corner point includes the steps of: A31. For the neighborhood window of the candidate corner point, calculate the pixel gradients in the x-axis and y-axis directions in the camera coordinate system through the Sobel operator 、 ; A32. For all pixels in the neighborhood window, calculate the weighted sum of the squared gradients, expressed as:

[0010] Wherein, represents the pixel gradient of the th pixel in the x-axis in the neighborhood window; represents the pixel gradient of the th pixel in the y-axis in the neighborhood window; represents the road sign point feature matrix; A33. Calculate the eigenvalues and of the matrix , and take the minimum value of and as the road sign point score of the candidate corner point; In step A33, the calculation method of the eigenvalue or is expressed by the following formula (1):

[0011] In formula (1), represents or ; represents solving the trace of the matrix , which is the sum of the main diagonal element values of the matrix , that is, the sum of and of the matrix ; represents the determinant of the matrix , is ; Preferably, in step S22, the method for representing the system state of the vehicle control device in the device body coordinate system by the inverse depth representation method includes the steps of: B1. For each of the road punctuation points selected from the same frame image, the inverse depth parameter value is calculated by the following formula (2):

[0012] In formula (2), represents the inverse depth parameter value of the road punctuation point; represents the straight-line distance from the road punctuation point to the position of the camera; B2. The system state of the vehicle control device in the device body coordinate system in each frame image that appears in consecutive frames is expressed by the following expression (3):

[0013] In expression (3), represents the position coordinates of the inertial navigation device installed in the vehicle control device, in the device body coordinate system ; represents the speed of the inertial navigation device, in the device body coordinate system ; represents the rotation attitude of the inertial navigation device, which is the mapping from the device body coordinate system to the world coordinate system ; represents the acceleration bias of the inertial navigation device, in the device body coordinate system ; represents the rotation bias of the inertial navigation device, in the device body coordinate system ; represents the translation of the vehicle control device from the camera coordinate system to the device body coordinate system, in the device body coordinate system ; represents the rotation from the camera coordinate system to the device body coordinate system ; represents the th direction vector of the road punctuation point in the image frame, in the camera coordinate system ; , being a natural number; represents the th inverse depth of the road punctuation point; It is calculated by the following formula (4):

[0014] In expression (4), represents the normalized coordinates of the th road punctuation point in the camera coordinate system;

[0015] and and respectively represent the horizontal axis coordinate, vertical axis coordinate and z-axis coordinate of the th road punctuation point in the camera coordinate system.

[0016] Preferably, step S3 specifically includes the steps of: S31. Estimate the pixel position of the road punctuation point in the th frame in the th frame in the image of the th frame according to the IMU data of the S32. Build a multi-layer image block pyramid for the th frame image, and calculate the photometric error at the corresponding position of the road punctuation point in the th frame in the th frame; S33. Update the prior system state of each road punctuation point in the th frame according to the calculated photometric error, and achieve an unbiased optimal estimate of the posterior state of each road punctuation point.

[0017] Preferably, step S31 includes the steps of: C1. Calculate the angle increment at time compared with time , velocity increment , and displacement increment ; C2. Predict the pixel coordinates of the road punctuation point in the first image frame collected at time and and in the second image frame collected at the current time according to the estimated state increments and

[0018] Preferably, in step C1, the specific calculation methods of the state increments and and are respectively expressed by the following formulas (6)-(8):

[0019] In formula (6), Indicates at moment and the th intermediate frame for which IMU data processing is not performed at each acquisition time point of the camera; Indicates at moment and the number of intermediate frames generated between moments; is the sampling interval duration of the IMU data; Indicates the angular velocity of the vehicle control device at the th intermediate frame;

[0020] In formula (7), Indicates at moment, the rotation matrix from the world coordinate system to the device body coordinate system; Indicates the gravitational acceleration, taking - ;

[0021] Preferably, step C2 specifically includes the steps: C21, set the spatial coordinates of a path of punctuation marks in the first image frame acquired at the moment to be , , , , moment, and estimate the spatial coordinates of this path of punctuation marks through the following formula (9):

[0022] C22, convert to pixel coordinates , and the conversion method is expressed by the following formula (10):

[0023] In formula (10), indicates the internal parameter matrix, which is used to implement the mapping from normalized coordinates to pixel coordinates;

[0024] respectively indicate the focal lengths of the camera in the x and y axis directions; respectively represent the coordinates of the center of the imaging plane on the x and y axes of the image plane.

[0025] Preferably, in step S32, the method for calculating the photometric error includes the steps of: D1, denote the predicted pixel coordinates of the th waypoint in the th frame as , and construct a multi-level image block pyramid at ; D2, solve the photometric error of the estimated waypoint in the th frame through the following formula (11):

[0026] In formula (11), represents the photometric error of all estimated waypoints at time for the th frame; represents the th waypoint; represents the number of estimated waypoints for the th frame; represents the th layer in the multi-level image block pyramid constructed in step D1; represents the number of levels of the multi-level image block pyramid; represents the th pixel in the image block in the multi-level image block pyramid; represents the valid pixel region in the image block; is the photometric error of the valid region of the th frame estimated th waypoint in the image block of the th layer of the pyramid; is calculated through the following formula (12):

[0027] In formula (12), represents the position of the th layer of the multi-level image block pyramid where the th pixel of the The pixel value at is at the position in the frame image where the road punctuation point at the current frame's pixel value; is the center point of the image block at the th layer of the pyramid of the frame image; is the relative position offset of the road punctuation point at the position compared to the center point; respectively represent the model parameters of the brightness change between two adjacent frames.

[0028] Preferably, in step D1, the method for constructing a multi-level image block pyramid at is as follows: Set the pyramid of the th frame image to be downsampled by a preset multiple, and then on each layer of the pyramid image, with as the center, select an image block of a preset size, and each layer of image blocks constitutes a multi-level image block pyramid; The preset multiple is 2 times, and the preset size is 8*8 pixels; The method for the optimal unbiased estimation of the posterior state of each road punctuation point in step S33 is: Construct the th frame's residual vector , expressed as follows:

[0029] With as the measurement, perform the update of the system state to obtain the posterior of the system state, including the pose information of the vehicle control device , represents the body coordinate position of the vehicle control device in the device body coordinate system, represents the attitude of the vehicle control device.

[0030] The present invention also provides a vehicle control device based on air trajectory recognition, which can implement the vehicle function control method based on air trajectory recognition as described above, including: A motion trajectory recognition mode trigger module, used to trigger the working mode of recognizing the motion trajectory of the vehicle control device; Tracking and positioning module, connected to the motion trajectory recognition mode triggering module, for performing real-time tracking and positioning on the moving vehicle control device through vision plus inertial navigation after being triggered to enter the working mode; Motion estimation and analysis module, connected to the tracking and positioning module, for using the real-time tracking and positioning result as the input of a pre-trained neural network model, and the model output is the motion trajectory analyzed for the vehicle control device; Function matching and execution module, connected to the motion estimation and analysis module, for matching the quasi-control function corresponding to the motion trajectory, and instructing the vehicle system to control the vehicle to execute the quasi-control function.

[0031] The present invention has the following beneficial effects: 1. Using a movable vehicle control device with mechanical buttons to control various functions of the vehicle, while retaining the tactile feedback, it increases the flexibility of vehicle control; 2. Triggering the trajectory recognition function with mechanical buttons, and controlling the vehicle to execute corresponding functions based on the recognized motion trajectory of the movable vehicle control device in the air, which increases the fun of vehicle control, and since it does not require complex multi-button operations and does not require operations in a gaze state like touch screen virtual buttons, it also increases the safety of vehicle control; 3. Based on vision-inertial sensors and neural networks to identify the motion trajectory of the movable vehicle control device in the air, this recognition method realizes centimeter-level recognition of the motion trajectory of the vehicle control device, ensures the accuracy of trajectory recognition, and thus improves the accuracy of vehicle function control based on the recognized motion trajectory. Brief Description of the Drawings

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly introduced below. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 It is a flowchart of the implementation steps of the vehicle function control method based on air trajectory recognition provided by the embodiment of the present invention; Figure 2 It is an example diagram of the device body coordinate system constructed in this embodiment; Figure 3 It is a schematic flowchart of the vision update method based on the multi-layer image block pyramid provided by this embodiment. Detailed Embodiments

[0034] The technical solutions of the present invention will be further described below in conjunction with the drawings and through specific embodiments.

[0035] Among them, the attached drawings are only for illustrative purposes, showing only schematic diagrams rather than physical diagrams, and should not be construed as a limitation on this patent; in order to better illustrate the embodiments of the present invention, some components in the attached drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the attached drawings may be omitted.

[0036] In the attached drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if terms such as "upper", "lower", "left", "right", "inner", "outer", etc. are used to indicate the orientation or positional relationship, it is based on the orientation or positional relationship shown in the attached drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms used to describe the positional relationship in the attached drawings are only for illustrative purposes and should not be construed as a limitation on this patent. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0037] In the description of the present invention, unless otherwise clearly specified and limited, if terms such as "connection" are used to indicate the connection relationship between components, this term should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] The vehicle function control method based on control trajectory recognition provided in this embodiment, as Figure 1 shown, its technical core includes: 1. Detect the holding state of the user on the movable vehicle control device through a mechanical button, and trigger the trajectory recognition mode of the vehicle control device. For example, after the user presses the mechanical button set on the movable vehicle control device, the vehicle control device sends an instruction to execute the trajectory recognition of the vehicle control device to the vehicle system through wireless communication such as Bluetooth. After receiving this instruction, the vehicle system enters the working mode of performing trajectory recognition on the vehicle control device.

[0039] 2. After entering the estimation recognition mode of the vehicle control device, perform centimeter-level precise positioning on the moving movable vehicle control device through visual recognition + inertial sensors, so as to ensure the accuracy of subsequent recognition of the movement trajectory of the vehicle control device.

[0040] 3. The positioning result of the vehicle control device in motion by visual recognition + inertial sensor is used as the input of the neural network model. The neural network model identifies and outputs the motion trajectory of the vehicle control device, then matches the quasi-control function corresponding to the motion trajectory, and finally the vehicle-mounted system controls the vehicle to execute the matched quasi-control function.

[0041] First, a brief description of how this embodiment triggers and enters the working mode for identifying the motion trajectory of the movable vehicle control device is as follows: For example, the vehicle control device generates a trigger command with the signal of the user pressing the physical button set on the vehicle control device, and sends the trigger command to the vehicle-mounted system. The vehicle-mounted system controls the visual recognition device set inside the vehicle and / or the inertial navigation device set in the vehicle control device to enter the working mode for identifying the motion trajectory of the vehicle control device. Or, the inertial navigation device directly enters the working mode for identifying the motion trajectory of the vehicle control device with the signal of the user pressing the physical button.

[0042] Next, an explanation of how this embodiment realizes the precise fusion positioning of the moving vehicle control device through visual recognition + inertial sensor is as follows: This embodiment proposes a lightweight visual-inertial fusion positioning method applicable to an embedded platform, with the goal of achieving high-precision positioning of the movable vehicle control device in the automotive cockpit scenario. Specifically, during the movement of the system (vehicle control device), the high-frequency acceleration and angular velocity data provided by the IMU (Inertial Measurement Unit) inertial navigation device are used to promote the propagation of the system state, and visual observation information is introduced in the system state update stage to update the system state.

[0043] The lightweight visual-inertial fusion positioning method provided by this embodiment has the following technical innovation points: 1. A smooth normalization and state parameterization method represented in the device body coordinate system are proposed, thereby reducing the uncertainty of the system state during propagation and reducing the error accumulation easily brought by continuous motion in the world coordinate system for the system state representation; 2. A robust visual update method based on a multi-layer image block pyramid is designed, which involves a fast key point coordinate prediction method, a robust method based on a multi-layer pyramid and coordinate optimization; 3. A body state estimation method based on the extended Kalman filter is designed, which combines visual and IMU positioning data to realize the real-time estimation of the position coordinates of the movable vehicle control device.

[0044] Next, a specific explanation of the first above-mentioned technical innovation point is as follows: In this embodiment, the coordinate system of the device body takes the installation position of the IMU inertial navigation device in the vehicle control device as the origin, and its axis is aligned with the inherent axis of the IMU inertial navigation device, as shown in Figure 2 the example in

[0045] The smoothing normalization and the state parameterization method represented in the coordinate system of the device body proposed in the first technical innovation point above specifically include the following steps: S21, construct the landmark feature matrix , and filter out the landmarks used to characterize the system state of the vehicle control device in the coordinate system of the device body in each frame of the continuous frames; the landmark is a three-dimensional space point with a preset detected feature in the vehicle cockpit environment; the preset detected feature is a three-dimensional space point with a significant feature and can be stably detected in the cockpit environment. Whether the three-dimensional space point has a significant feature and can be stably detected is obtained by extracting FAST corner points from the continuous image frames and calculating the scores of each FAST corner point through S, and screening among each FAST corner point. The FAST corner point is used to detect the area where the local pixels in the image change significantly. The specific method for obtaining the landmark by screening FAST focus points in the image includes the steps: A1, select a pixel point in each frame of the continuous frames, and define the brightness of the pixel point as , then take the pixel point as the center, and select (preferably 3 pixels) pixels on the circle with a radius of (preferably 12 pixels); A2, judge the number of pixel points with brightness greater than or brightness less than among the (preferably 12 pixels) pixel points selected in step A1, whether it is greater than a preset number threshold , if so, extract the pixel point as a candidate corner point and transfer to step A3; if not, do not take the pixel point as a candidate corner point; A3, return to step A1, loop through all pixel points in the same frame of the image, and after completing the loop through, calculate the landmark scores for all candidate corner points extracted from the same frame of the image; In this embodiment, the method for calculating the landmark scores of candidate corner points includes the steps: A31, for the neighborhood window of the candidate corner point, calculate the pixel gradients , in the x-axis and y-axis directions in the camera coordinate system through the Sobel operator; Take as an example, the method for calculating the pixel gradient by the Sobel operator is briefly described as follows: The Sobel operator in the x direction is:

[0046] The Sobel operator in the y direction is:

[0047] For the th pixel in the neighborhood window, a 3*3 pixel block centered on this pixel is selected , and convolved with or respectively to obtain the pixel gradients in the x-axis and y-axis directions , :

[0048]

[0049] The size of the neighborhood window of the candidate corner points is preferably 15×15 pixels.

[0050] A32. For all pixels in the neighborhood window, calculate the weighted sum of the squared gradients, expressed as:

[0051] Among them, represents the pixel gradient of the th pixel in the x-axis of the neighborhood window; represents the pixel gradient of the th pixel in the y-axis of the neighborhood window; represents the road sign feature matrix; A33. Calculate the eigenvalues and of the matrix , and take the minimum value of and as the road sign score of the candidate corner point. The higher the score, the more significant the texture change at the position where the candidate corner point is located.

[0052] In this embodiment, the eigenvalues and are calculated by the following formula (1):

[0053] In formula (1), represents or ; Denote the trace of the solution matrix which is the sum of the values of the main diagonal elements of the matrix , that is, the above ; Denote the determinant of the matrix which is for ; A4. Select the top candidate corner points sorted from high to low score as the various punctuation marks filtered for this frame of image.

[0054] After filtering out the punctuation marks for each frame in the continuous frames through step S21, proceed to the step: S22. According to the various punctuation marks filtered in step S21, represent the system state of the vehicle control device in the device body coordinate system through the inverse depth representation method. The representation method specifically includes the steps: B1. For the various punctuation marks filtered from the same frame of image, calculate the inverse depth parameter value through the following formula (2):

[0055] In formula (2), represents the inverse depth parameter value of the punctuation mark; represents the straight-line distance (depth distance) from the punctuation mark to the position where the camera is located; The representation method significantly improves the numerical stability of depth estimation in the initialization stage by converting depth information into an unbounded positive definite parameter, and avoids the optimization divergence problem that is prone to occur in traditional forward depth representation when lacking prior information.

[0056] B2. Express the system state of the vehicle control device in each frame of the continuous frames in the device body coordinate system through the following expression (3) :

[0057] In expression (3), represents the seat coordinates of the inertial navigation device (IMU inertial measurement unit) installed in the vehicle control device, which is in the device body coordinate system ; represents the speed of the inertial navigation device, which is in the device body coordinate system ; represents the rotation attitude of the inertial navigation device, which is from the device body coordinate system to the world coordinate system Mapping; Indicates the acceleration bias of the inertial navigation device, in the device body coordinate system Under; Indicates the rotation bias of the inertial navigation device, in the device body coordinate system Under; Indicates the translation from the camera coordinate system to the device body coordinate system of the vehicle control device, in the device body coordinate system Under; Indicates from the camera coordinate system To the device body coordinate system Rotation; Indicates the direction vector of the th road marking point in the image frame, in the camera coordinate system Under, , being a natural number; Indicates the th inverse depth of the road marking point.

[0058] Calculated by the following formula (4):

[0059] In expression (4), Indicates the th normalized coordinate of the road marking point in the camera coordinate system;

[0060] , , Respectively indicate the horizontal axis coordinate, vertical axis coordinate and z-axis coordinate of the th road marking point in the camera coordinate system.

[0061] The three-dimensional coordinates of the road marking point are parameterized into a direction vector through formulas (4)-(5).

[0062] The following specifically describes a robust vision update method based on a multi-layer image block pyramid proposed in the above second technological innovation point.

[0063] A vision update method based on a multi-layer image pyramid, used to estimate the position of each frame of image in the vehicle control device in consecutive frames. When the th frame in the consecutive frames arrives, through the IMU data of the th frame, the The road punctuation of the frame is at the pixel position in the frame image. Subsequently, by constructing a multi-layer image block pyramid for the frame image, calculate the photometric error at the corresponding position of the road punctuation in the frame; finally, according to the calculated photometric error, update the prior system state of each road punctuation in the frame to achieve an unbiased optimal estimate of the posterior state of each road punctuation. The implementation process of the visual update method based on the multi-layer image pyramid is as

[0064] shown, and the specific implementation steps are as follows: Figure 3 1. After the system is powered on and running, at the 0th frame, the system reads the detected IMU data and image frame data. At this time, due to the lack of historical data, the state estimation framework cannot run, and the 0th frame is the initialization frame of the system. 1. When the system starts up and runs, at the 0th frame, the system reads the detected IMU data and image frame data. At this time, since there is no historical data, the state estimation framework cannot run, and the 0th frame is the initialization frame of the system.

[0065] 2. Starting from the 1st frame. The continuously running system will continuously execute the EKF-based fusion positioning, repeat the prediction-update process of the Kalman filter, and iteratively update the system state. Specifically: Assume that at the frame, the system receives new image data and IMU data. Since the detection frequency of the IMU inertial measurement unit is greater than the image acquisition frequency of the camera, there are a certain number of unprocessed IMU data intermediate frames between the moment and the moment. For these IMU data intermediate frames, obtain the angular increment compared with the moment, velocity increment , and displacement increment by integration. The specific calculation methods are expressed by the following formulas (6)-(8) respectively:

[0066] In formula (6),

[0067]

[0067]

[0067] represents the angular velocity of the vehicle control device at the th intermediate frame;

[0068] In formula (7), represents the rotation matrix from the world coordinate system to the device body coordinate system (IMU coordinate system) at moment.

[0069] is a loop iteration variable. At the initial moment, the initial rotation matrix from the world coordinate system to the device body coordinate system is determined through calibration , and at subsequent moments, there is:

[0070] represents the change in the rotation attitude of the device body coordinate system from the kth moment to the (k + 1)th moment, that is, the rotation attitude at the next moment is the superposition of the rotation increment on the previous moment's attitude. Here, is different from the predicted increment obtained from the above IMU data , needs to be updated and determined by visual measurement for the overall system state in subsequent steps.

[0071] represents the gravitational acceleration, taking -9.8 N / kg;

[0072] 3. Based on the calculated state increment , , , predict the pixel coordinates of the road landmark in the first image frame collected at moment in the second image frame collected at the current moment, specifically: Let the spatial coordinates (in the world coordinate system) of a road landmark in the first image frame collected at moment be , are the horizontal axis coordinate value, vertical axis coordinate value, and z-axis coordinate value of this road landmark respectively; Then, according to the state increment , , estimated in the above steps, and estimate the spatial coordinates of this road landmark at

[0073] Then, convert to pixel coordinates , the conversion method is expressed by the following formula (10):

[0074] In formula (10), represents the internal parameter matrix, which is used to realize the mapping from normalized coordinates to pixel coordinates;

[0075] respectively represent the focal lengths of the camera in the x and y axis directions; respectively represent the coordinates of the center of the imaging plane on the x and y axes of the image plane; The internal parameter matrix is confirmed by pre-calibrating the camera or the camera's factory configuration.

[0076] At the same time, due to the true pixel coordinates of the road punctuation at the previous moment to the true space coordinates is known, and it is:

[0077] Combined with the conversion relationship from the spatial coordinates of the road punctuation at the moment to the predicted pixel coordinates at the moment, the conversion relationship from the pixel coordinates of the road punctuation at the moment to the can be simplified to:

[0078] represents the process described in detail above, starting from the pixel coordinates of a certain road punctuation at the k-1 moment, calculating the spatial coordinates at the k-1 moment in sequence, estimating the spatial coordinates at the k moment, and then calculating the estimated pixel coordinates corresponding to the k moment.

[0079] The following specifically describes the method for calculating the photometric error of the estimated road punctuation in the frame: In this embodiment, by constructing a multi-layer image block pyramid for the frame image, the photometric error of the estimated road punctuation in the frame is calculated. The photometric error is used to reflect the accuracy of the predicted coordinates. Denote the in the The pixel coordinates of a road punctuation predicted in the frame are . In theory, the pixel value at should be exactly the same as the corresponding pixel coordinates in the frame. However, in practice, there is a certain error between the two. The calculation method of photometric error specifically includes the following steps: (1), Construct a multi-level image patch pyramid at . Preferably, the pyramid is set to 2-fold downsampling, that is, for the original image (the 0th layer), perform multiple 2-fold downsamplings, and select an 8*8 image patch around to obtain a multi-level image patch pyramid.

[0080] Specifically, the multi-level image pyramid with 2-fold downsampling means that both the image width and height of the layer pyramid are reduced to half of those of the layer. The value of the downsampled pixel is the pixel mean of the corresponding 2*2 image patch (the 2*2 image patch centered on this pixel) in the upper layer. After constructing the image pyramid, on each layer of the image, centered on , select an 8*8 image patch. The image patches of each layer are combined to form a multi-level image patch pyramid. The multi-level image features help improve the robustness of the system. (2), Solve the photometric error of the road punctuation estimated in the frame through the following formula (11):

[0081] In formula (11), represents the photometric error of all road punctuations estimated at the moment for the frame; represents the th road punctuation; represents the number of road punctuations estimated for the frame; represents the th layer in the multi-level image patch pyramid constructed in step D1; represents the number of levels of the multi-level image patch pyramid; represents the th pixel in the image patch (preferably 8*8 pixel size) in the multi-level image patch pyramid; Indicates the area where the pixel in the image block is located. If the coordinates of the road punctuation are (i, j), then ; For the th road punctuation estimated in the th layer of the pyramid in the image block, the photometric error of the valid area.

[0082] It is calculated by the following formula (12):

[0083] In formula (12), Indicates the pixel value at the position where the th pixel in the image block of the th layer of the multi-level image block pyramid of the th frame image is located; is the pixel value at the position of the road punctuation in the th frame image at the position in the current th frame; is the center point of the image block of the th layer of the pyramid of the th frame image; is the relative position offset of the road punctuation at the position compared with the center point;

[0084] 4. Visual update Based on the photometric error of each road punctuation, construct the residual vector of the th frame, expressed as follows:

[0085] Taking as the measurement, perform an update on the system state to obtain the posterior of the system state, which includes the pose information of the vehicle control device , indicating the body coordinate position of the vehicle control device in the body coordinate system of the device, indicating the attitude of the vehicle control device.

[0086] The process of newly performing the update of the system state is expressed by the following formula:

[0087] where is the Kalman gain at time k, which is used to adjust the influence of the residual on the state update.

[0088] The following specifically elaborates on the process of the body state estimation method based on the extended Kalman filter provided in this embodiment, which combines visual and IMU positioning data to achieve real-time estimation of the position coordinates of a movable vehicle control device: There are many existing methods for predicting the motion trajectory of a target object through a neural network model according to the real-time positioning and tracking results of the target object. In this embodiment, a lightweight convolutional neural network is constructed to identify the classification and recognition of the motion trajectory of the vehicle control device. And according to the pre-set association relationship between the motion trajectory and the vehicle control function, the associated vehicle control function is matched and executed, so as to realize the corresponding control function of the vehicle according to the motion trajectory. The technical implementation process is as follows: 1.1 The input data of the trajectory recognition model of the lightweight convolutional neural network is the tracking and positioning information of the device within a preset time interval. It is assumed that within the time interval for completing an effective action, there are frames of trajectory data, then the input data , where the position information of the device is represented by the coordinates obtained by positioning the vehicle control device, and the attitude information of the device is represented by a rotation quaternion. Then the input data containing one effective action is a (1 + 3 + 4) * matrix. "1" refers to the time between two consecutive tracking and positioning positions, "3" represents the 3 coordinate values of the three-dimensional space coordinates, and "4" represents the rotation quaternion. Preferably, .

[0089] 1.2 Normalize the estimated position information. When identifying the motion of the trajectory, the relative motion trend between different frames is concerned, rather than the absolute position of the device in space, that is, in the world coordinate system. Therefore, it is hoped that the parameters of the model focus on the relative motion between the trajectory points. Through this local normalization method, the model can better learn the relative spatial position characteristics between the trajectory points. Specifically, for the position information input for each frame, perform secondary normalization. The " " in the position information represents the th frame.

[0090] First, globally normalize the position information through the following formula:

[0091] For the normalization result of the coordinate position of the vehicle control device in the frame;

[0092] For the coordinate information that has completed global normalization, with as the origin, perform local normalization on the input position information, expressed as:

[0093] 1.3 Construct a trajectory classification model. This model is a lightweight convolutional neural network, which includes a convolutional layer with 32 channels and a size of 4*14, and passes through a BN normalization layer and a ReLU activation function at one time, and the size of the output feature is 32*3*4; a convolutional layer with 64 channels and a size of 2*5, a BN normalization layer and a ReLU activation function, a max pooling layer MaxPool, and the size of the output feature is 64*2*2; subsequently, perform a flattening (Flatten) operation on the feature to convert the feature into a one-dimensional vector, and the output is 256*1; pass through a fully connected layer with an input of 256 and an output of 128; pass through a fully connected layer with an input of 128 and an output of 64; finally pass through a fully connected layer with an input of 64 and an output of N, and finally obtain a one-dimensional vector with a length of N, where N is the number of preset gesture types. The design of this model aims to efficiently extract the temporal and spatial features in the input trajectory through a lightweight convolutional neural network and achieve the trajectory classification task.

[0094] The first-layer convolution captures the dynamic changes of the trajectory in the time dimension and the global features in the spatial dimension with a larger window (4*14). Next, further extract the local temporal and spatial patterns through a smaller convolutional kernel (2*5), and cooperate with the BN normalization and ReLU non-linear activation functions to enhance the feature expression ability. The max pooling layer compresses the size of the feature map through downsampling, reduces the computational complexity while retaining important features. Subsequently, the third convolutional kernel (1*4) focuses on the extraction of detailed features in the spatial dimension to further enhance the classification performance. Finally, through Flatten and fully connected layers, the features extracted by convolution are integrated into a one-dimensional vector, and the probability that the trajectory belongs to each classification is output. The overall model structure is simple and efficient, adapts to low-computation resource scenarios, and can capture the comprehensive features of the trajectory in time and space, suitable for the task of trajectory classification.

[0095] In this embodiment, the method for creating a trajectory classification training set and training the model is briefly described as follows: The specific method for creating the training set is: Collect trajectory data over a period of time through a vehicle control device. Each trajectory contains multiple time frames, and each frame records the corresponding position information of the device. Ensure that the collected trajectory data covers all target classification categories and is as diverse as possible (trajectories under different speeds, directions, and environmental conditions).

[0096] Data preprocessing: Normalize the collected data according to the global normalization and local normalization methods introduced above.

[0097] Denoising: Perform smoothing filtering on the trajectory data to remove noise and improve the model's focusing ability on trajectory features.

[0098] Data augmentation: Augment the data by rotation, translation, scaling, or time perturbation to increase the diversity of trajectories and improve the robustness of the model.

[0099] Subsequently, label and partition the collected data. Manually label the corresponding trajectory category for each trajectory data to generate classification labels.

[0100] Partition the dataset into a training set, a validation set, and a test set (e.g., in an 8:1:1 ratio), ensuring that the distribution of each category in different datasets is balanced to avoid classification bias.

[0101] After completing the collection of the dataset, use the created training set to train the model. Preferably, select the cross-entropy loss function as the optimization objective for the classification task; use the Adam optimizer or the SGD optimizer, combined with a learning rate scheduling strategy to improve the training efficiency; input the processed trajectory data into the model, use the backpropagation algorithm to update the model parameters, and at the same time evaluate the model performance on the validation set to prevent overfitting. After training is completed, evaluate the model performance on the test set.

[0102] Initialization and operation of the model. When the system is running continuously, the device will first collect a certain amount of historical data frames. Specifically, it is necessary to wait for at least N frames of data. When the historical data accumulates to N frames, the device starts to process the input data in a sliding window manner. For each newly input frame, the device combines the previous N frames of historical data to construct a new input feature sequence. This feature sequence is input into the classification model, and the model generates the corresponding trajectory recognition result through inference and outputs the unique identifier (ID) of the trajectory category. This result ID can be further used for functional responses of vehicle control or other application scenarios.

[0103] Formulate the vehicle control functions corresponding to the motion trajectories. Specifically, according to the predefined functional requirements, construct a mapping table to associate the trajectory ID output by the classification model with the corresponding control functions one by one. When the trajectory ID output by the model is recognized, the system can quickly retrieve the corresponding control logic from the mapping table and execute the corresponding functional operations.

[0104] This embodiment also provides a vehicle control device based on air trajectory recognition, which can implement the above-mentioned vehicle function control method based on air trajectory recognition. The device includes: A motion trajectory recognition mode triggering module, which is used to trigger the working mode of recognizing the motion trajectory of the vehicle control device; A tracking and positioning module, connected to the motion trajectory recognition mode triggering module, which is used to perform real-time tracking and positioning on the moving vehicle control device through vision plus inertial navigation after triggering and entering the working mode; A motion estimation and analysis module, connected to the tracking and positioning module, which is used to take the real-time tracking and positioning result as the input of a pre-trained neural network model, and the model outputs the analyzed motion trajectory of the vehicle control device; A function matching and execution module, connected to the motion estimation and analysis module, which is used to match the quasi-control function corresponding to the motion trajectory, and instruct the vehicle machine system to control the vehicle to execute the quasi-control function.

[0105] Specifically, the vehicle control device realizes efficient communication between the device and the vehicle machine terminal through Bluetooth Low Energy (BLE) broadcast technology. The innovation of this mechanism lies in combining air trajectory recognition with the control logic of the vehicle machine terminal, and realizing real-time and accurate function triggering through wireless communication means, providing a more intelligent and convenient interaction method for the cockpit. The following are the specific implementation schemes: Hardware system composition: The function triggering mechanism of the present invention is implemented by the following modules: A trajectory recognition device: including a monocular camera, a MEMS IMU, an SoC, and a BLE communication module, which is used to sense the air trajectory input by the user, classify the trajectory in real time, and send the result to the vehicle machine terminal through a BLE broadcast signal.

[0106] The vehicle machine terminal: including a BLE receiving module and a control logic module, which is used to receive the broadcast signal sent by the trajectory recognition device and execute the corresponding control function according to the preset trajectory function mapping table.

[0107] Function triggering process: The specific implementation process of function triggering is divided into the following steps: (1) Input and recognition of air trajectory The user inputs an air trajectory through gestures, and the trajectory recognition device collects trajectory motion data through built-in sensors. The device uses a sliding window mechanism to construct features for continuous N-frame historical data and the current newly input frame, and inputs the feature sequence into a trajectory classification model deployed on an embedded platform. The classification model outputs the corresponding trajectory category ID according to the input sequence.

[0108] (2) Trajectory validity judgment To ensure the stability and reliability of the system, the trajectory recognition device determines the validity of the trajectory category ID output by the model. Preferably, it includes: Determine whether the trajectory belongs to a predefined set of valid trajectories; exclude noise; exclude invalid trajectories caused by misoperations.

[0109] When the trajectory category is determined to be valid, trigger the subsequent signal sending mechanism.

[0110] (3)Transmission of BLE broadcast signals The built-in BLE communication module of the trajectory recognition device is activated and sends a control signal corresponding to the trajectory category in broadcast form. Preferably, the composition of the broadcast information frame includes: [trajectory category ID, timestamp information, device unique identifier (Device ID), used for differentiation among multiple communication devices, other additional information (such as specific scenario mode identifier, etc.), frame header and frame tail of the BLE frame] (4)Signal reception and processing at the in-vehicle unit The BLE receiving module at the in-vehicle unit continuously listens for the broadcast signals sent by the trajectory recognition device. When the in-vehicle unit receives a broadcast signal, it parses the trajectory category ID and other information therein, and retrieves the corresponding control function according to the preset trajectory function mapping table.

[0111] (5)Function execution The in-vehicle unit calls the corresponding control logic module according to the parsed control function type and executes function operations. These include controlling the opening and closing of the window, seat control, volume adjustment, ambient light adjustment, etc.

[0112] It should be noted that the above specific embodiments are only the preferred embodiments of the present invention and the applied technical principles. Those skilled in the art should understand that various modifications, equivalent replacements, changes, etc. can be made to the present invention. However, as long as these transformations do not deviate from the spirit of the present invention, they should be within the protection scope of the present invention. Additionally, some terms used in the specification and claims of this application are not restrictive, but are only for the convenience of description.

Claims

1. A vehicle function control method based on air trajectory recognition, characterized in that: Includes steps: S1, triggering to enter a working mode for identifying the motion trajectory of the vehicle control device; S2, real-time tracking and positioning of the vehicle control device in motion by vision and inertial navigation; S3, using the real-time tracking and positioning result of step S2 as the input of the pre-trained neural network model, and the model outputs the motion trajectory analyzed for the vehicle control device; S4, matching the simulated control function corresponding to the motion trajectory, and instructing the vehicle computer system to control the vehicle to execute the simulated control function.

2. The vehicle function control method based on air trajectory recognition according to claim 1 is characterized in that: In step S1, the method of triggering the entry into the working mode of identifying the motion trajectory of the movable vehicle control device is: The vehicle control device generates a trigger instruction using a signal that a user presses a physical button set on the vehicle control device, and sends the trigger instruction to the vehicle system. The vehicle system controls the visual recognition device and / or the inertial navigation device set in the vehicle control device to enter a working mode for motion trajectory recognition of the vehicle control device, or the inertial navigation device directly uses the signal that the user presses the physical button to enter a working mode for motion trajectory recognition of the vehicle control device.

3. The vehicle function control method based on air trajectory recognition according to claim 1 is characterized in that: Step S2 specifically includes the following steps: S21, construct landmark feature matrix , selecting a landmark point for representing the system state of the vehicle control device in the device body coordinate system in each of the continuous frames, wherein the landmark point is a three-dimensional space point having a preset detected feature in the vehicle cabin environment; S22, based on the landmarks screened out in step S21, and using an inverse depth characterization method, representing the system state of the vehicle control device in the device body coordinate system.

4. The vehicle function control method based on air trajectory recognition according to claim 3 is characterized in that: In step S21, the method for selecting landmark points from each frame of image specifically comprises the following steps: A1, select pixel points from each frame of the continuous frame , and define the pixel point The brightness is , and then take the pixel point As the center of the circle, select the radius on the circle pixels; A2, determine the selected The brightness of the pixels is greater than or brightness is less than The number of pixels Is it greater than the preset quantity threshold? If so, extract the pixel point Take it as a candidate corner point and proceed to step A3; If not, the pixel point is not as the candidate corner point; A3, returning to step A1, looping through all pixel points in the same frame image, and after completing the looping, calculating the landmark point scores for all the candidate corner points extracted from the same frame image; A4, select the top scores in descending order The candidate corner points are used as the landmark points screened for the frame image.

5. The vehicle function control method based on air trajectory recognition according to claim 4 is characterized in that: In step A1, is 3 pixels, There are 16 pixels and the 16 pixels are evenly arranged around the circumference; The quantity threshold in step A2 is 12; In step A3, the method for calculating the landmark point score of the candidate corner point comprises the steps of: A31, for the neighborhood window of the candidate corner point, the pixel gradient in the x-axis and y-axis directions in the camera coordinate system is calculated using the Sobel operator , ; A32, for all pixels in the neighborhood window, calculate the weighted sum of the squared gradients, expressed as: in, Indicates the first The pixel gradient of pixels on the x-axis; Indicates the first The pixel gradient on the y-axis of pixels; Represents the landmark point feature matrix; A33, calculate the matrix The eigenvalue of and , and and The minimum value of is taken as the landmark point score of the candidate corner point; In step A33, the eigenvalue or The calculation method of is expressed by the following formula (1): In formula (1), express or ; Represents the solution matrix The trace of is the matrix The sum of the main diagonal element values ​​of is the matrix of and of and; Representation Matrix The determinant of for .

6. The vehicle function control method based on air trajectory recognition according to any one of claims 3 to 5, characterized in that: In step S22, the method for representing the system state of the vehicle control device in the device body coordinate system by the inverse depth characterization method comprises the following steps: B1, for each landmark point selected from the same frame image, the inverse depth parameter value is calculated by the following formula (2): In formula (2), Indicates the inverse depth parameter value of the landmark point; Indicates the straight-line distance from the landmark point to the camera position; B2, the system state of the vehicle control device in the device body coordinate system appearing in each frame image in the continuous frames is expressed by the following expression (3): In expression (3), Indicates the position coordinates of the inertial navigation device installed in the vehicle control device, expressed in the device body coordinate system Down; Indicates the speed of the inertial navigation device, expressed in the device body coordinate system Down; Indicates the rotational attitude of the inertial navigation device, which is from the device body coordinate system To world coordinate system The mapping of Represents the acceleration bias of the inertial navigation device, expressed in the device body coordinate system Down; Indicates the rotation offset of the inertial navigation device, expressed in the device body coordinate system Down; represents the translation of the vehicle control device from the camera coordinate system to the device body coordinate system, and represents the translation of the vehicle control device from the camera coordinate system to the device body coordinate system Down; Represents the coordinate system from the camera To the device body coordinate system Rotation of Indicates the first The direction vector of the landmark point, expressed in the camera coordinate system Down, , is a natural number; Indicates The inverse depth of each landmark point; It is calculated by the following formula (4): In expression (4), Indicates The normalized coordinates of landmark points in the camera coordinate system; , , Respectively represent The horizontal axis coordinate, vertical axis coordinate and z axis coordinate of each landmark point in the camera coordinate system.

7. The vehicle function control method based on air trajectory recognition according to any one of claims 1 to 5, characterized in that: Step S3 specifically includes the following steps: S31, according to the first Frame IMU data, estimate the The landmark point in the frame is Pixel position in the frame image; S32, by Frame image builds a multi-layer image block pyramid and calculates The landmark point in the frame is Photometric error of the corresponding position in the frame; S33, based on the calculated photometric error, update the The prior system state of each landmark point in the frame is used to achieve an unbiased optimal estimate of the posterior state of each landmark point.

8. The vehicle function control method based on air trajectory recognition according to claim 7 is characterized in that: Step S31 includes the steps of: C1, calculation Compare to time Angle increment at time , speed increment , displacement increment ; C2, based on the estimated state increment , , ,predict The landmark point in the first image frame collected at the moment is The pixel coordinates in the second image frame acquired at time instant.

9. The vehicle function control method based on air trajectory recognition according to claim 8, characterized in that: In step C1, the state increment , , The specific calculation methods are expressed by the following formulas (6)-(8): In formula (6), Indicated in Moment and The first time when the IMU data is not processed at each camera acquisition time point Intermediate frames; Indicated in Moment and The number of intermediate frames generated between moments; It is the sampling interval of IMU data; Indicates that the vehicle control device is in the Angular velocity of the intermediate frames; In formula (7), Indicated in At this moment, the rotation matrix from the world coordinate system to the device body coordinate system; represents the acceleration due to gravity, take - ; 。 10. The vehicle function control method based on air trajectory recognition according to claim 8, characterized in that: Step C2 specifically includes the steps of: C21, set The spatial coordinates of a punctuation point in the first image frame collected at time , are the horizontal axis coordinate value, vertical axis coordinate value and z axis coordinate value of the landmark point respectively, according to the state increment estimated in step C1 , , , and estimate the landmark point at Spatial coordinates of time : C22, will Convert to pixel coordinates , the conversion method is expressed by the following formula (10): In formula (10), Represents the internal parameter matrix, which is used to realize the mapping from normalized coordinates to pixel coordinates; Respectively represent the focal length of the camera in the x and y axis directions; They represent the coordinates of the center of the imaging plane on the x and y axes of the image plane respectively.

11. The vehicle function control method based on air trajectory recognition according to claim 10, characterized in that: In step S32, the method for calculating the photometric error comprises the steps of: D1, remember the correct answer in The first The landmark point is The predicted pixel coordinates in the frame are ,exist Construct a multi-level image block pyramid; D2, solve the following formula (11) The photometric error of the estimated landmark points in the frame: In formula (11), Indicated in Time to The photometric errors of all landmark points estimated in the frame; Indicates waypoints; Indicates The number of landmarks estimated in the frame; represents the first image block in the multi-level image block pyramid constructed in step D1 layer; represents the number of levels of the multi-level image block pyramid; represents the first image block in the multi-level image block pyramid pixels; Represents the valid pixel area in the image block; For the Frame estimation The landmark point is on the pyramid Photometric error of valid areas in image patches in a layer; It is calculated by the following formula (12): In formula (12), express The frame image is in the first level of the multi-level image block pyramid. The first image block in the layer The position of the pixel The pixel value at ; is The position in the frame image The landmark point at the current The pixel values ​​in the frame; for The first pyramid of the frame image The center point of the image patch of the layer; is at the stated location The relative position offset of the landmark point at the position compared to the center point; They respectively represent the model parameters of the brightness change between two adjacent frames.

12. The vehicle function control method based on air trajectory recognition according to claim 11, characterized in that: In step D1, The method for constructing a multi-level image block pyramid is: For The frame image pyramid is set to a preset multiple downsampling, and then on each layer of the pyramid image, As the center, an image block of a preset size is selected, and each layer of image blocks constitutes a multi-level image block pyramid; The preset multiple is 2 times, and the preset size is 8*8 pixels; The method for optimal unbiased estimation of the posterior state of each landmark point in step S33 is: Build The residual vector of the frame , expressed as follows: by To update the system state, we need to obtain the a posteriori , Contains the position information of the vehicle control device , represents the body coordinate position of the vehicle control device in the device body coordinate system, Indicates the posture of the vehicle control device.

13. A vehicle control device based on air trajectory recognition, which can implement the vehicle function control method based on air trajectory recognition as claimed in any one of claims 1 to 12, characterized in that: include: A motion trajectory recognition mode trigger module, used to trigger the entry into a working mode for recognizing the motion trajectory of the vehicle control device; A tracking and positioning module, connected to the motion trajectory recognition mode triggering module, is used to perform real-time tracking and positioning of the moving vehicle control device through vision and inertial navigation after triggering to enter the working mode; A motion estimation and analysis module, connected to the tracking and positioning module, used to use the real-time tracking and positioning results as input to a pre-trained neural network model, and the model outputs the motion trajectory analyzed for the vehicle control device; The function matching and execution module is connected to the motion estimation and analysis module, and is used to match the simulated control function corresponding to the motion trajectory, and instruct the vehicle system to control the vehicle to execute the simulated control function.

Images