Suspension control method, device, equipment and medium

Abstract

The invention relates to a suspension control method and device, equipment and a medium, and the suspension control method comprises the steps: analyzing a main body causing picture change based on an image picture, and determining a motion mode corresponding to the main body, the main body comprising at least one of imaging equipment, a moving object in the picture and an environment; and switching the motion mode to a motion mode corresponding to the main body so as to control the suspension to perform corresponding motion along with the motion of the main body. By implementing the suspension control method provided by the invention, the problem that an existing suspension control method cannot be directly suitable for a simulation scene and is difficult to fully simulate real motion, so that the immersion feeling of a user is insufficient can be solved.

Description

Technical Field

[0001] This application relates to the technical field of suspension control, and particularly to a suspension control method, device, equipment and medium. Background Art

[0002] The suspension system has become a key technology for improving the comfort and stability of the cockpit. Existing suspension control technologies are mainly used to adjust the suspension during vehicle driving, aiming to adapt to different road conditions and thus improve the smoothness and comfort of the vehicle.

[0003] With the continuous improvement of the requirements for user experience in industries such as automobiles, virtual reality, and simulators, the application scenarios of immersive viewing and virtual reality (VR) are becoming increasingly rich.

[0004] However, the current suspension control methods cannot be directly applied to simulation scenarios, especially for realistic simulation scenarios with complex movements, making it difficult to fully simulate real movements and resulting in insufficient immersion for users in simulation scenarios. Summary of the Invention

[0005] Based on this, this application provides a suspension control method, device, equipment and medium, which can improve the problem that the existing suspension control methods cannot be directly applied to simulation scenarios and are difficult to fully simulate real movements, thus resulting in insufficient immersion for users.

[0006] In a first aspect, this application provides a suspension control method, which includes: analyzing the subject that causes the change of the image frame based on the image frame, and determining the motion mode corresponding to the subject, where the subject includes at least one of an imaging device, a moving object in the frame, and the environment; switching the motion mode to the motion mode corresponding to the subject to control the suspension to perform corresponding movements following the movement of the subject.

[0007] In combination with the first aspect, in the first possible implementation manner of the first aspect, the steps of analyzing the subject that causes the change of the image frame based on the image frame and determining the motion mode corresponding to the subject include: analyzing the subject that causes the change of the image frame based on the image frame; if the analysis result is one of the subjects of a moving object, an imaging device, and the environment, the motion modes corresponding to the respective subjects are a fixed perspective mode, a moving perspective mode, and a dynamic background mode; if the analysis result is an imaging device and a moving object, the corresponding motion mode is a mixed perspective mode; if the analysis result is no subject, the corresponding motion mode is a non-compliance mode or a quick-switching mode, where the suspension is prohibited from moving in the non-compliance mode and the quick-switching mode.

[0008] In combination with the first aspect or the first possible implementation method of the first aspect, in the second possible implementation method of the first aspect, the step of analyzing the subject causing the picture change based on the image screen includes: detecting the proportion of active pixels in the image screen, and the area with maximum energy in the image screen; if the proportion of active pixels in the image screen is less than the corresponding threshold, there is an area with maximum energy, and there is violent motion in the area with maximum energy, then it is determined that the subject causing the picture change is a moving object.

[0009] In combination with the first aspect or the first possible implementation method of the first aspect, in the third possible implementation method of the first aspect, the step of analyzing the subject causing the picture change based on the image picture includes: detecting the proportion of active pixels in the image picture, and the degree of consistency of the optical flow direction of the image picture; if the proportion of active pixels in the image picture is greater than the corresponding threshold value, and the degree of consistency of the optical flow direction of the image picture is greater than the corresponding threshold value, then determining that the subject causing the picture change is the imaging device.

[0010] In combination with the first aspect or the first possible implementation method of the first aspect, in the fourth possible implementation method of the first aspect, the step of analyzing the subject causing the picture change based on the image screen includes: detecting the proportion of active pixels in the image screen, and the global motion and local motion of the image screen; if the proportion of active pixels in the image screen belongs to the corresponding preset interval, the significance of the global motion is greater than the corresponding threshold and the richness of the local motion is greater than the corresponding threshold, then determining that the subject causing the picture change is the imaging device and the active object.

[0011] In combination with the first aspect or the first possible implementation method of the first aspect, in the fifth possible implementation method of the first aspect, the step of analyzing the subject causing the picture change based on the image picture includes: detecting the degree of motion fluctuation of the background area of ​​the image picture, and the significance of periodic motion; if the degree of motion fluctuation of the background area of ​​the image picture is less than the corresponding threshold and the significance of periodic motion of the background area is greater than the corresponding threshold, then determining that the subject causing the picture change is the environment.

[0012] In combination with the first aspect or the first possible implementation method of the first aspect, in the sixth possible implementation method of the first aspect, the step of determining the subject that causes the picture change based on image picture analysis includes: detecting edge density or edge density change rate, and detecting the global average displacement or the proportion of active pixels; if the edge density or the edge density change rate is greater than the corresponding threshold value, it is determined that the picture has changed suddenly and there is no subject, wherein, in the case of a sudden change in the picture, no subject corresponds to a fast switching mode; if the global average displacement or the proportion of active pixels is less than the corresponding threshold value, it is determined that the picture change is not significant and there is no subject, wherein, in the case of an insignificant picture change, no subject corresponds to a non-standard mode.

[0013] In a second aspect, the present application further provides a suspension control device, which includes: an analysis unit configured to analyze a subject that causes a change in the image frame based on the image frame and determine a motion mode corresponding to the subject, where the subject includes at least one of an imaging device, a moving object in the frame, and the environment; and a control unit configured to switch the motion mode to the motion mode corresponding to the subject to control the suspension to perform corresponding motions following the motion of the subject.

[0014] In combination with the second aspect, in the first implementable manner of the second aspect, the aforementioned analysis unit is specifically configured to: analyze a subject that causes a change in the image frame based on the image frame; if the analysis result is one of a moving object, an imaging device, and the environment as the subject, the corresponding motion modes for each subject are a fixed perspective mode, a moving perspective mode, and a dynamic background mode respectively; if the analysis result is an imaging device and a moving object, the corresponding motion mode is a mixed perspective mode; if the analysis result is no subject, the corresponding motion mode is a non-compliance mode or a fast switching mode, where the suspension is prohibited from moving in the non-compliance mode and the fast switching mode.

[0015] In combination with the second aspect or the first implementable manner of the second aspect, in the second implementable manner of the second aspect, the aforementioned analysis unit is specifically configured to: detect the proportion of active pixels in the image frame and the region with the maximum energy in the image frame; if the proportion of active pixels in the image frame is less than the corresponding threshold, there is a region with the maximum energy, and there is violent motion in the region with the maximum energy, then determine that the subject that causes the change in the image frame is a moving object.

[0016] In combination with the second aspect or the first implementable manner of the second aspect, in the third implementable manner of the second aspect, the aforementioned analysis unit is specifically configured to: detect the proportion of active pixels in the image frame and the degree of consistency of the optical flow direction in the image frame; if the proportion of active pixels in the image frame is greater than the corresponding threshold and the degree of consistency of the optical flow direction in the image frame is greater than the corresponding threshold, then determine that the subject that causes the change in the image frame is an imaging device.

[0017] In combination with the second aspect or the first implementable manner of the second aspect, in the fourth implementable manner of the second aspect, the aforementioned analysis unit is specifically configured to: detect the proportion of active pixels in the image frame, and the global motion and local motion of the image frame; if the proportion of active pixels in the image frame belongs to the corresponding preset interval, the significance degree of the global motion is greater than the corresponding threshold and the richness degree of the local motion is greater than the corresponding threshold, then determine that the subject that causes the change in the image frame is an imaging device and a moving object.

[0018] Combined with the second aspect or the first implementable manner of the second aspect, in the fifth implementable manner of the second aspect, the foregoing parsing unit is specifically configured to: detect the degree of motion fluctuation of the background area of the image frame and the significance of periodic motion; if the degree of motion fluctuation of the background area of the image frame is less than the corresponding threshold and the significance of the periodic motion of the background area is greater than the corresponding threshold, determine that the main body causing the change in the frame is the environment.

[0019] Combined with the second aspect or the first implementable manner of the second aspect, in the sixth implementable manner of the second aspect, the foregoing parsing unit is specifically configured to: detect the edge density or the edge density transformation rate, and detect the global average displacement or the proportion of active pixels; if the edge density or the edge density transformation rate is greater than the corresponding threshold, determine that the frame has undergone a sudden change and there is no main body, where, in the case of a sudden change in the frame, no main body corresponds to the fast switching mode; if the global average displacement or the proportion of active pixels is less than the corresponding threshold, determine that the change in the frame is not significant and there is no main body, where, in the case of a non-significant change in the frame, no main body corresponds to the non-compliance mode.

[0020] In a third aspect, the present application further provides a suspension control device, which includes a processor and a memory, and the processor and the memory are connected through a bus; the processor is configured to execute multiple instructions; the memory is configured to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor to perform the suspension control method according to the first aspect or any implementable manner of the first aspect.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are adapted to be loaded and executed by the processor to perform the suspension control method according to the first aspect or any implementable manner of the first aspect.

[0022] In summary, the present application provides a suspension control method, device, equipment, and storage medium. In order to enable users to obtain a real motion experience in the image frame, the present application first parses out the main body that actually causes the change in the frame, and then switches the motion mode to the motion mode corresponding to the main body, so as to control the suspension to perform corresponding motions following the motion of the main body, making the motion feedback in the cockpit / simulator highly synchronized with the motion changes in the image frame. This feedback can accurately simulate the user's viewing experience, enhance the immersion of the user in watching movies in the cockpit, make the user experience more natural and real, rather than simply controlling the suspension motion according to the overall change of the image frame, and without excessive manual intervention, generally improving the simulation effect and simulation efficiency. This technology can be widely applied to fields such as automobiles, simulators, virtual reality, and reality simulation, and can especially meet the requirements of real-time simulation of real-scene images and enhance the immersion of users. Description of the Drawings

[0023] Figure 1 Schematic flow chart of a suspension control method according to an embodiment of the present application;

[0024] Figure 2 Schematic block diagram of a suspension control device according to an embodiment of the present application;

[0025] Figure 3 Structural block diagram of a suspension control device according to an embodiment of the present application. Detailed implementation manners

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] The existing suspension technology mainly obtains the dynamic information of the vehicle body through sensors (such as accelerometers, gyroscopes, etc.), and adjusts the state of the suspension according to this information to adapt to different road conditions, thereby improving the smoothness and comfort of the vehicle.

[0028] On the one hand, since the existing suspension control is mainly applied to improve the ride comfort, for example, by reducing the spring stiffness and shock absorber damping to weaken the bumpy feeling, while the suspension control of the present application is used for simulation scenarios, for example, by increasing the spring stiffness and shock absorber damping to highlight the bumpy feeling, the existing suspension control technology cannot be directly applied to simulation scenarios;

[0029] On the other hand, the existing suspension control technology mainly relies on sensor detection, because the motion detected by the sensor can most directly reflect the real motion situation. Even if image detection is used, it is only used as an auxiliary detection means and there is no need to deeply analyze and classify the motion in the picture, because in the real motion scenario, the main consideration is the environmental interaction, that is, the motion feeling brought to the user by the physical dynamic changes of the vehicle itself. At this time, the motion of the imaging device can be determined only by analyzing the overall change of the picture generated by the imaging device, because the imaging device is fixed on the vehicle and the motion of the imaging device reflects the motion of the vehicle.

[0030] Different from the environmental interaction, in the simulation scenario, the main consideration is the visual experience, that is, the vehicle, other moving objects, the environment, etc. will all bring visual motion experience to the user. Because it is not enough to only consider the vehicle motion in the simulation scenario, the real motion feeling of the user in the simulation scenario not only comes from the vehicle motion, but also comes from the motion of other subjects. By deeply analyzing at least one subject motion in the image frame and controlling the suspension motion according to at least one subject motion, the immersion of the reality simulation can be greatly improved, but this does not need to be considered when improving the ride comfort in the real driving scenario.

[0031] In addition, the usual reality simulation technology does not analyze the video images deeply and comprehensively enough. It relies on manual post-production to add a motion control process that matches the images, and most of them are only applicable to the reality simulation of computer-generated imagery (CGI). It is difficult to meet the reality simulation of live-action images because, compared with CGI, live-action images have more complex and diverse motions and motion details. Therefore, the existing reality simulation has the disadvantages of poor simulation effects and being extremely time-consuming and laborious.

[0032] Based on this, the present application provides a suspension control method. First, the main body that actually causes the change of the image is analyzed, and then the motion mode is switched to the motion mode corresponding to the main body to control the suspension to perform corresponding motions following the motion of the main body, rather than simply controlling the suspension motion according to the overall change of the video image. This enables the user to obtain a real motion experience when in the environment of the image. It should be noted that the suspension control method provided by the present application is applicable to the simulation of any type of image, especially suitable for the reality simulation of live-action images, because it can deeply and comprehensively analyze the complex motions in live-action images, automatically switch to the appropriate motion mode, simulate the real motion experience of the user when in the environment of the live-action image, without excessive manual intervention, which is very time-saving and laborious, and greatly improves the simulation effect and simulation efficiency.

[0033] Specifically, the present application analyzes the video image to obtain the main body that causes the change of the image. The main body includes at least one of an imaging device, a moving object in the image, and the environment. According to the difference of the main body that causes the change of the image, it switches to different motion modes. For example, the motion modes include at least one of a fixed perspective mode, a moving perspective mode, a mixed motion mode, a non-compliance mode, a dynamic background mode, and a fast switching mode. When the main body is a moving object in the image, it switches to the fixed perspective mode; when the main body is the imaging device, it switches to the moving perspective mode; when the main body is the imaging device and a moving object in the image, it switches to the mixed motion mode; when the target moving main body is the environment in the image, it switches to the dynamic background mode; when the change of the image is not significant (regarded as no main body), it switches to the non-compliance mode; when the image changes suddenly (regarded as no main body), it switches to the fast switching mode.

[0034] Control the suspension to move according to the suspension motion strategy in the target motion mode, that is, control the suspension to perform corresponding motions following the motion of the main body. When there is no main body, the suspension motion is prohibited. The definitions of each motion mode and the suspension motion strategies in each motion mode can refer to Table 1 below.

[0035]

[0036]

[0037] Table 1, Six motion modes

[0038] It should be noted that the suspension mentioned in this application is mainly an active suspension (Active Suspension), such as an active air suspension (Active Air Suspension), a hydraulic active suspension (Hydraulic Active Suspension), an electromagnetic active suspension (Electromagnetic Active Suspension), etc. Parameters such as the damping, stiffness, or height of the suspension can be actively adjusted, and effects such as left - right tilt and up - down vibration can be achieved through parameter control. Compared with the passive suspension, it can accurately simulate the real motion effect.

[0039] The suspension control method provided in this application can be applied to a suspension control device or a suspension control equipment. The suspension control device and the suspension control equipment can be in - vehicle terminals, controllers, processors, electronic control units (Electronic Control Unit, ECU), etc. that can implement computing functions and suspension control functions. In addition, the suspension control device and the suspension control equipment can perform data interaction with other servers, terminal devices, controllers, processors, ECU, etc., and execute the method proposed in this application.

[0040] For a better understanding of the suspension control method proposed in this application, this application also provides an embodiment of the suspension control method, as Figure 1 shown. Next, taking the suspension device as the execution subject, this method will be described in detail:

[0041] 100: Analyze the subject that causes the change in the image frame based on the image frame, and determine the motion mode corresponding to the subject;

[0042] 200: Switch the motion mode to the motion mode corresponding to the subject to control the suspension to perform corresponding motions following the motion of the subject.

[0043] In step 100, for different subjects, the corresponding motion modes are different. The subject can be any one or any combination of an imaging device, a moving object in the frame, and the environment. Correspondingly, any one or any combination can respectively correspond to a motion mode. By following different subjects in different motion modes, this application can flexibly adapt to different image contents and scene changes through multi - mode motion response. This enables the suspension control device to make accurate responses to various situations such as imaging device motion, moving object motion, and environmental motion, effectively improving the limitation of the traditional method that can only respond to a single motion type, and realizing diversified and precise control of the suspension under different subjects.

[0044] In practical applications, parsing all combinations of multiple subjects is extremely time-consuming and laborious, and it will also make the subsequent suspension control very complex, resulting in some unnecessary movements, reducing the user experience, and there is still room for improvement in terms of real-time performance and parsing efficiency.

[0045] In order to improve the parsing efficiency and real-time performance in the suspension control simulation scenario and enhance the user experience, this application parses several common situations that cause changes in the picture and provides corresponding motion modes. Specifically, the step 100 of parsing the subject that causes the picture change based on the image picture and determining the corresponding motion mode of the subject includes:

[0046] 110: Parse the subject that causes the picture change based on the image picture;

[0047] 120: If the parsing result is one of the subjects of the moving object, imaging device, and environment, the corresponding motion modes of each subject are the fixed perspective mode, the moving perspective mode, and the dynamic background mode respectively;

[0048] 130: If the parsing result is the imaging device and the moving object, the corresponding motion mode is the mixed perspective mode;

[0049] 140: If the parsing result is no subject, the corresponding motion mode is the non-compliance mode or the fast switching mode. Among them, the suspension is prohibited from moving in the non-compliance mode and the fast switching mode.

[0050] Among them, the motion modes include the fixed perspective mode, the moving perspective mode, the mixed motion mode, the non-compliance mode, the dynamic background mode, and the fast switching mode. When the subject that causes the picture change is a moving object, the corresponding motion mode is the fixed perspective mode; when the subject that causes the picture change is the moving perspective mode, the corresponding motion mode is the imaging device; when the subject that causes the picture change is the imaging device and the moving object, the corresponding motion mode is the mixed motion mode; when there is no subject that causes the picture change (the picture change is not significant), the corresponding motion mode is the non-compliance mode; when the subject that causes the picture change is the environment, the corresponding motion mode is the dynamic background mode; when there is no subject that causes the picture change (the picture suddenly changes), the corresponding motion mode is the fast switching mode.

[0051] In step 110, when the suspension control device parses the subject that causes the picture change, it can determine whether the subject that causes the picture change is at least one of the imaging device, the moving object in the picture, and the environment through deep learning methods, etc. However, most algorithms for in-depth image analysis such as deep learning are relatively complex. In this regard, this application also proposes to use a simple image analysis method for subject parsing, and the image change situation of the image picture can be determined through image analysis techniques such as the pixel value method and optical flow analysis.

[0052] For example, the change value of the pixel value of each pixel point of the image frame is calculated by the pixel value method, or the proportion of active pixels in the image frame is calculated by the optical flow technique, and then the main body causing the change of the frame is analyzed according to the magnitude of the change value of the pixel value or the proportion of active pixels. Among them, the change value of the pixel value and the proportion of active pixels are used to represent the degree of change of the image frame. The change values of the pixel values and the proportions of active pixels of the frames caused by different main body movements are usually different. For example, the change values of the pixel values and the proportions of active pixels of the frame caused by the movement of the imaging device are usually greater than the change values of the pixel values and the proportions of active pixels of the frame caused by the movement of the moving object, respectively.

[0053] It should be noted that although the optical flow analysis technique, as an important technique in the field of computer vision, is applied in multiple fields, this application innovatively applies the optical flow analysis technique specifically to the analysis of the moving main body to realize the application of the suspension control technique in the simulation scenario. Moreover, compared with other analysis techniques such as the pixel value method, the optical flow analysis technique performs better and can more accurately identify global motion and / or local motion because the optical flow analysis technique realizes refined motion analysis. By calculating the motion vectors of pixel points, it can distinguish different types of motion, while the pixel value method cannot distinguish whether the change is caused by light change or main body movement; it has excellent anti-interference ability, can filter out light fluctuations and noise interference, and avoid misjudgment, while the pixel value method is easily affected by the environment; it has spatio-temporal continuity, and can identify the complex motion of the main body through temporal correlation analysis to facilitate accurate control of the suspension motion according to the motion of the main body after switching the motion mode subsequently. Generally speaking, the optical flow is closer to the human visual perception mechanism of motion. By adopting the optical flow analysis technique for main body analysis in this application, the analysis accuracy is improved, thereby enhancing the authenticity of the simulation and the real motion experience and immersion of the user in the simulation scenario.

[0054] The steps of determining the proportion of active pixels in the image frame by the optical flow technique described above include: calculating the displacement vector field of each pixel in the image frame, calculating the global average displacement and displacement standard deviation according to the displacement vector field of each pixel; calculating the activity threshold according to the global average displacement and displacement standard deviation; determining the pixels with a displacement vector field greater than the threshold as active pixels, and determining the pixels with a displacement vector field less than or equal to the threshold as inactive pixels; using the activity region mask to represent the region where the active pixels with a displacement vector field greater than the activity threshold are located; and calculating the ratio of the number of active pixels to the number of global pixels according to the activity region mask to obtain the proportion of active pixels.

[0055] Among them, the displacement vector field F(x, y) of each pixel, F(x, y) can be expressed as [u(x, y), v(x, y)], or can be expressed as [u, v], and can be obtained by minimizing the following energy function E(u, v) using the TV-L1 optical flow algorithm:

[0056] Let \(W\) be the window size. For example, \(W = 15\times15\) pixels. The solution method is the dual optimization algorithm, and the number of iterations \(N\) iter \(= 100\), \(u\) is the horizontal displacement, \(v\) is the vertical displacement, and \(I\) t is the image frame at time \(t\), and \(I\) t+1 is the image frame at time \(t + 1\). \((x,y)\) represents the position of each pixel in \(I\) t , and \((x + u,y + v)\) represents the position of each pixel in \(I\) t+1 . and are the spatial gradients of the displacement field, \(\lambda\) represents the smoothing intensity. For example, \(\lambda = 0.05\), or \(\lambda\) is dynamically adjusted. \(\lambda_0\) is the initial preset value of \(\lambda\), and \(\mu\) A is the global average displacement.

[0057] Global average displacement \(N\) is the total number of pixels in the global area, and the displacement amplitude field \(u(x,y)\) is the horizontal displacement of the pixel \((x,y)\), \(v(x,y)\) is the vertical displacement of the pixel \((x,y)\), and the displacement standard deviation

[0058] The activity threshold \(\tau\) flow \(=\mu\) A \(+ 3\sigma\) A , where \(\mu\) A is the global average displacement, and \(\sigma\) A is the displacement standard deviation;

[0059] The active pixel \(A(x,y)>\tau\) flow , and the inactive pixel \(A(x,y)\leq\tau\) flow ;

[0060] The active area mask \(M\) active \(=\{(x,y)|A(x,y)>\tau\) flow \}\), and the active area mask \(M\) active is used to identify the pixel area with significant motion in the image frame.

[0061] The proportion of active pixels \(|M|\) active | / N, where \(|M|\) active | is the number of active pixels, and \(N\) is the total number of pixels in the global area.

[0062] In the process of analyzing the main body causing the change of the picture in step 100 and step 110, in order to further improve the accuracy while ensuring real-time performance and analysis efficiency, the present application also proposes corresponding specific analysis strategies for analyzing various main bodies. Specifically:

[0063] The first parsing strategy, the steps of parsing the subject that causes the screen change based on the image screen include: detecting the proportion of active pixels in the image screen and the region R with the maximum energy in the image screen i , where N is the total number of global pixels, M active is the active region mask, |M active | is the number of active pixels, is the proportion of active pixels, R i is the region with the maximum energy; if the proportion of active pixels in the image screen is less than the corresponding threshold m1, there is a region R with the maximum energy i , and there is violent movement in the region R with the maximum energy i , for example and μ A (R i )>n1μ A , then it is determined that the subject causing the screen change is an active object, where m1 is the threshold for determining whether the movement of the image screen is local movement, simply referred to as the first threshold, m1 is a constant less than 1, μ A (R i ) is the average displacement of the region with the maximum energy, μ A is the global average displacement, n1 is the violent multiple, and n1 is a constant greater than 1, μ A (R i )>n1μ A means that the average displacement μ i of the region R with the maximum energy A (R i ) is greater than the global average displacement μ A of the violent multiple n1. In other words, the main motion energy of the region with the maximum energy is significantly higher than the background noise, so it is considered that there is violent movement in the region with the maximum energy. In addition, in practical applications and experiments, m1 = 0.15 and n1 = 3 are preferred parameters. Using these preferred parameters can achieve a good parsing effect and accurately parse whether the subject causing the screen change is an active object.

[0064] In the first parsing strategy, before determining that there is violent movement in the region R with the maximum energy i , the displacement amplitude field A(x, y) can be divided into at least one region by the MeanShift clustering algorithm, and the region with the maximum energy is selected from the at least one region to obtain the region R with the maximum energy i , and the average displacement μ i (R A ) of the region R with the maximum energy is calculated, where the average displacement μ i of the region R with the maximum energy i and the average displacement μ A (Ri ) It is used to measure the intensity of local motion; if the average displacement μ i of the maximum energy region R A (R i ) is greater than the product of the global average displacement μ A and the intensity multiple n1. In other words, if the average displacement μ i of the maximum energy region R A (R i ) exceeds the global average displacement μ A of the intensity multiple n1, it is determined that local motion, the maximum energy region, and the maximum energy region R i have intense motion.

[0065] The second parsing strategy. The steps of parsing the main body causing the change in the image frame based on the above include: detecting the proportion of active pixels in the image frame and the degree of consistency of the optical flow direction in the image frame where N is the total number of global pixels, M active is the active area mask, |M active | is the number of active pixels, is the proportion of active pixels, is the degree of consistency of the optical flow direction, μ A is the global average displacement, Consistency is the direction consistency, (x, y) is the pixel coordinate, θ(x, y) is the pixel optical flow direction angle, is the global average direction angle; if the proportion of active pixels in the image frame is greater than the corresponding threshold m2, and the degree of consistency of the optical flow direction in the image frame is greater than the corresponding threshold, for example and then it is determined that the main body causing the change in the image frame is the imaging device. Among them, when , it means that the proportion of active pixels is high, the overall image frame moves, global motion is detected, m2 is the threshold for determining whether the motion in the image frame is global motion, simply referred to as the second threshold, and m2 is a constant less than 1, n2 is the threshold for determining whether the optical flow direction in the image frame is consistent. When , it means that the optical flow direction is consistent, excluding the interference of independent motion. In addition, in practical applications and experiments, m2 = 0.85, n2 = 0.8 are preferred parameters. Using these preferred parameters can achieve a good parsing effect and accurately parse whether the main body causing the change in the image frame is the imaging device.

[0066] The third parsing strategy. The steps of parsing the main body causing the change in the image frame based on the above include: detecting the proportion of active pixels in the image frame and the global motion L and local motion S of the image frame, where the global motion L and local motion S can be obtained through RPCA (Robust Principal Component Analysis) optical flow decomposition, N is the number of global pixels, M active is the active region mask, |M active | is the number of active pixels, is the active pixel ratio; if the proportion of active pixels in the image frame belongs to the corresponding preset interval [m3, m4], the significance level of the global motion L is greater than the corresponding threshold and the richness of the local motion S is greater than the corresponding threshold, for example and ||L|| F > n3 and ||S||0 > n4N, then it is determined that the main bodies causing the change of the frame are the imaging device and the moving object, where [m3, m4] is the preset interval for determining whether the motion of the image frame is the global motion L and the local motion S, m3 and m4 are constant values, m3 is the upper limit value for determining whether the motion of the image frame is the global motion L and the local motion S, referred to as the third threshold, m4 is the lower limit value for determining whether the motion of the image frame is the global motion L and the local motion S, referred to as the fourth threshold, when it means that the proportion of active pixels is medium and the imaging device and the moving object coexist; ||L|| F is the Frobenius norm (i.e., the Frobenius norm) of the global motion L, ||L|| F is used to represent the significance level of the global motion, ||S||0 is the number of pixels of the local motion S, ||S||0 is used to represent the richness of the local motion, n3 and n4 are constant values, n3 is the threshold for determining whether the global motion L is significant, n4 is the threshold for determining whether the local motion S is rich, when ||L|| F > n3, it means that the significance level of the global motion is greater than the corresponding threshold and the imaging device moves itself, when ||S||0 > n4N, it means that the richness of the local motion is greater than the corresponding threshold and there are multiple independent moving bodies. In addition, in practical applications and experiments, when m3 = 0.3, when m4 = 0.8, n3 = 15, n4 = 0.15 are preferred parameters, and using these preferred parameters can achieve a good parsing effect and accurately parse whether the main bodies causing the change of the frame are the imaging device and the moving object.

[0067] In the third parsing strategy, the present application can accurately separate global motion and local motion through the analysis of optical flow features in the image by the optical flow algorithm and the Robust Principal Component Analysis (RPCA) technology. This enables the suspension control device to more precisely identify the motion conditions of the imaging device and the moving object, including the motion direction and amplitude, etc., and control the suspension motion according to the superposition of the local motion and the global motion control signals, providing richer and more delicate cockpit feedback.

[0068] Fourth parsing strategy. The foregoing steps of analyzing the main body causing the change in the image frame include: detecting the degree of motion fluctuation of the background area of the image frame and the significance level of the periodic motion where, A background is the optical flow displacement amplitude of the background area, Var(A background ) represents the variance of the optical flow displacement amplitude of the background area, μ A is the global average displacement, which is used to measure the degree of motion fluctuation of the background area of the image frame, E max is the peak energy of the Fast Fourier Transform (FFT) of the background area, that is, the energy of the most significant frequency component in the background area. E max reflects the energy level of the most significant frequency component (i.e., the periodic signal component) in the background area in the frequency domain, which is obtained by taking the maximum energy value in the frequency domain after performing the Fourier transform on the background area. E is the total energy of the background area, which is obtained by taking the sum of the energies of all frequency components after performing the Fourier transform on the background area. E reflects the overall total energy level of the background area in the frequency domain. The significance level of the periodic motion is measured by representing the energy ratio of the periodic signal component in the entire signal; if the degree of motion fluctuation of the background area of the image frame is less than the corresponding threshold n5 and the significance level of the periodic motion of the background area is greater than the corresponding threshold n6, for example and then it is determined that the main body causing the change in the frame is the environment. When , it indicates that the background motion variance is low, excluding random noise. When When it indicates that the energy proportion of the periodic signal component in the entire signal is relatively large, it is considered that there is a significant periodic motion. n5 is the threshold for determining whether the motion in the background area fluctuates greatly, and n6 is the threshold for determining whether the periodic motion in the background area is significant. n5 and n6 are constants less than 1. Additionally, in practical applications and experiments, n5 = 0.1 and n6 = 0.7 are preferred parameters. Using these preferred parameters can achieve a good analysis effect and accurately analyze whether the main body causing the change in the picture is the environment.

[0069] The fifth analysis strategy. The steps for analyzing the main body causing the change in the picture based on the image frame described above include: detecting the edge density or the Edge Density Change Rate (EDCR). Here, N represents the total number of pixels, F t is the optical flow field at time t, F t-1 is the optical flow field at time t - 1, F t -F t-1 is the optical flow field difference, ||F t -F t-1 ||F is the F-norm of the optical flow field difference. is the edge density, EDCR is the edge density change rate, and the edge density can be obtained through edge detection and calculation using the Canny algorithm; if the edge density or the edge density change rate is greater than the corresponding threshold, for example or EDCR > n8, it is determined that the picture has undergone a sudden change and there is no main body. Here, σ F is the standard deviation. ΔF = F t -F t-1 , ΔF = [Δu, Δv], Δu represents the horizontal displacement of the pixel point in the horizontal direction (x-axis), and Δv represents the vertical displacement of the pixel point in the vertical direction (y-axis). μ A is the global average displacement. In the case of a sudden change in the picture, no main body corresponds to the fast switching mode. When it indicates that the optical flow field difference is greater than n7 times the average fluctuation level. At this time, there is a sudden change in the image frame caused by lens splicing or rapid panning. When EDCR > n8, it indicates that the edge density has changed significantly. At this time, there is a sudden change in the image frame caused by scene transformation. n7 is the threshold for determining whether the edge density of the image frame is too large, and n8 is the threshold for determining whether the edge density change rate of the image frame is too large. n7 and n8 are constant values. Generally speaking, a sudden change in the picture means that the image frame has changed due to visual transitions such as lens splicing, rapid panning, or scene transformation. Additionally, in practical applications and experiments, n7 = 4 and n8 = 40% are preferred parameters. Using these preferred parameters can achieve a good analysis effect and accurately analyze whether the picture has undergone a sudden change.

[0070] The sixth analysis strategy, the above-mentioned steps of analyzing the subject causing the picture change based on the image picture include: detecting the global average displacement μ A Or active pixel ratio Among them, N is the number of global pixels, M active is the active region mask, |M active | is the number of active pixels, is the ratio of active pixels; if the global average displacement μ A Less than the corresponding threshold n9, or the percentage of active pixels is smaller than the corresponding threshold m5, for example or μ A <n9,则确定画面变化不显著,此时无主体,其中,当 or μ A <n9时,表示运动特征微弱,画面接近静止或画面幅度过小,此时认为未检测到显著运动,并且在画面变化不显著情况下,无主体对应未达标模式,m5和n9为常数值,m5和n9为用于判定影像画面的变化是否不显著的阈值,m5为第五阈值。另外,在实际应用和实验中,m5=0.05,n9=2为优选参数,采用该优选参数可以实现很好的解析效果,准确解析出画面的变化是否显著。

[0071] In one practicable manner, the aforementioned parsing strategies can be executed in parallel:

[0072] In the case of large changes in the picture, at least two of the following analysis results may appear: sudden changes in the picture, the subject causing the picture change is the environment, the imaging device and the moving object, and the imaging device. In order to improve the real movement experience, the priority of the above-mentioned analysis results decreases in turn, and accordingly, the priority of the fast switching mode, the dynamic background mode, the mixed viewing angle mode, and the mobile viewing angle mode decreases in turn. This is because: because too violent movement will reduce the user experience, the fast switching mode is higher than the other modes to limit unnecessary movement of the suspension; although the consistency of the optical flow direction of the picture changes caused by the environmental movement is not as good as that of the imaging device, there is also a certain consistency of the optical flow direction. According to the analysis strategy of the imaging device (i.e., the second analysis strategy), the picture changes caused by the environmental movement may be mistaken for the picture changes caused by the movement of the imaging device. Therefore, when it is analyzed that the subject causing the picture change includes the environment and the imaging device at the same time, it is switched to the dynamic background mode in advance to improve the accuracy of the analysis and the authenticity of the simulation; when it is analyzed that the subject causing the picture change includes the imaging device and the moving object at the same time, it is switched to the mixed viewing angle mode in advance to obtain a rich movement experience;

[0073] In the case of small changes in the screen, two parsing results may occur: the main body causing the screen change is a moving object and the screen change is not significant. To enhance the real motion experience, the priorities of the foregoing parsing results decrease in sequence. Correspondingly, the priority of the fixed perspective mode is higher than that of the non-compliance mode. This is because according to the parsing strategy for detecting significant screen motion (i.e., the sixth parsing strategy), the screen change caused by a moving object may be misinterpreted as a screen change caused by environmental noise, etc. Therefore, when the parsing results include both insignificant screen motion and a moving object as the main body causing the screen change, the fixed perspective mode is pre-switched to improve the accuracy of parsing and the authenticity of simulation.

[0074] In another feasible way, the foregoing parsing strategies are executed in a certain order to enhance the real-time performance of main body parsing and suspension control:

[0075] 111: Detect the global motion and local motion in the image screen. For example, detect the global motion and local motion in the image screen by the proportion of active pixels in the image screen or the global average displacement. If then it indicates that there is local motion in the screen. If then it indicates that there is global motion in the screen. If then it indicates that there is both local motion and global motion in the screen. If then it indicates that there is no global motion and local motion, where m5 ≤ m0 < m1 ≤ m3 < m4 ≤ m2;

[0076] 112: If there is no global motion and local motion, it is determined that the screen change is not significant and there is no main body;

[0077] 113: If there is only global motion, analyze whether the screen has mutated, whether the main body causing the screen change is the imaging device, and whether the main body causing the screen change is the environment. For example, if or EDCR > n8, it is determined that the screen has mutated and there is no main body. For another example, if then it is determined that the main body causing the screen change is the imaging device. For another example, if and then it is determined that the main body causing the screen change is the environment;

[0078] 114: If there is only local motion, analyze whether the main body causing the screen change is a moving object; for example, if and μ A (R i ) > n1μ A , then it is determined that the main body causing the screen change is a moving object;

[0079] 115: If there is global motion and local motion, analyze whether the main body causing the change in the picture is the imaging device and the moving object. For example, if ||L|| F > n3 and ||S||0 > n4N, it is determined that the main body causing the change in the picture is the imaging device and the moving object;

[0080] Among them, for the meanings of the parameters of the formula in the foregoing example and the preferred parameters, reference can be made to the foregoing first parsing strategy to the sixth parsing strategy, which will not be elaborated here.

[0081] For step 200, after the motion mode is switched to the target motion mode, the suspension control device focuses on the main body causing the change in the picture in the target motion mode, and controls the suspension to perform corresponding motions according to the motion of the main body causing the change in the picture.

[0082] In an implementable manner, when performing suspension control, the suspension control device can generate a suspension control command according to at least one of the motion amplitude, motion frequency, and motion intensity of the main body. For example, control the suspension to perform translation, tilt, and lift according to the motion amplitude of the main body, control the frequency of the periodic motion of the suspension according to the motion frequency of the main body, and control the motion intensity of the suspension according to the motion intensity of the main body, etc.

[0083] In another implementable manner, the suspension control device controls the suspension to follow the motion of the main body to perform corresponding motions according to different suspension control rules in different modes, so that the suspension is more adapted to the motion characteristics of the corresponding main body in different motion modes, improving the user's real motion experience and immersion feeling in the simulation. Specifically:

[0084] First, in the fixed perspective mode, switch the motion mode to the fixed perspective mode, generate a suspension control signal according to the centroid displacement and the change amount of motion energy of the moving object, and control the suspension motion according to the suspension control signal:

[0085] 211: Perform target detection and target tracking on the moving object in the video image;

[0086] Among them, the suspension control device can perform target detection on the moving object in the video image through YOLOv8, etc., and output the bounding box of the moving object. For example, the bounding box Bi of the i-th target i =(x i ,y i ,w i ,h i ), x i represents the x coordinate (starting position in the horizontal direction) of the upper left corner of the i-th detection box, y i represents the y coordinate (starting position in the vertical direction) of the upper left corner of the i-th detection box, w irepresents the width (horizontal extension length) of the i-th detection box, h i represents the height (vertical extension length) of the i-th detection box. Additionally, the moving object can be continuously tracked through the DeepSORT algorithm to generate the trajectory of the i-th moving object t represents the moment, where t = 1, 2,..., n, and n is an integer greater than 2. (x t , y t ) represents the position of the i-th moving object at the moment t;

[0087] 212: Calculate the centroid displacement of the moving object by performing optical flow calculation on the bounding box area including the moving object, and calculate the motion energy of the moving object;

[0088] Among them, the suspension control device calculates the optical flow field F i of each pixel inside the bounding box B of the i-th moving object i = [u i (x, y), v i (x, y)], where u i (x, y) represents the horizontal displacement of each pixel of the i-th moving object in the horizontal direction (x-axis), and v i (x, y) represents the vertical displacement of each pixel of the i-th moving object in the vertical direction (y-axis), and calculates the displacement mean value of all pixels inside the bounding box B i based on the optical flow field of each pixel inside the bounding box B, and calculates the change amount (Δx i , Δy i ) of the centroid displacement of the i-th moving object according to the displacement mean value. Δx represents the horizontal displacement of the centroid of the i-th moving object in the horizontal direction (x-axis), and Δy represents the vertical displacement of the centroid of the i-th moving object in the vertical direction (y-axis); calculate the motion energy of the i-th moving object i ; u i That is, u i (x, y), v i That is, v i (x, y) and calculate the change amount ΔE of the motion energy of the i-th moving object according to the motion energy E i of the i-th moving object i ;

[0089] Furthermore, in the case where there are multiple moving objects in the image frame, the motion energy E i of each moving object and the area |B iAssign weights, and weighted-average the change in the centroid displacement and the change in the motion energy of all active objects according to the assigned weights, that is, assign the weights of each active object according to the area of the detection frame of each active object and the motion energy of each active object, weighted-average the change in the centroid displacement of each active object according to the weights of each active object, and weighted-average the change in the motion energy of each active object according to the weights of each active object;

[0090] The weight assignment logic for assigning the weights of each active object according to the area of the detection frame of each active object and the motion energy of each active object includes:

[0091] w i = αE i + β|B i |, where w i is the weight of the i-th active object, E i is the motion energy of the i-th active object, |B i | is the area of the detection frame of the i-th active object, and α and β are balance coefficients, the magnitudes of which can be adjusted according to the specific scenario. This weight assignment logic indicates that active objects with more intense motion and larger area are more important;

[0092] The weighted-average calculation process for weighted-averaging the change in the centroid displacement of each active object according to the weights of each active object and weighted-averaging the change in the motion energy of each active object according to the weights of each active object includes:

[0093] Weighted-average of the change in the centroid displacement: Among them, is the weighted-average change in the centroid displacement, representing the comprehensive motion direction of all active objects in the image frame, N represents the number of active objects in the image frame, w i is the weight of the i-th active object, (Δx i , Δy i ) is the change in the centroid displacement of the i-th active object, and j represents the total number of active objects in the image frame;

[0094] Weighted-average of the change in the motion energy: is the weighted-average change in the motion energy, representing the comprehensive motion intensity of all active objects in the image frame, w i is the weight of the i-th active object, ΔE i is the change in the motion energy of the i-th active object, and j represents the total number of active objects in the image frame.

[0095] 213: Generate a suspension control signal based on the change in the centroid displacement and the change in the motion energy of the moving object, and control the suspension movement according to the suspension control signal;

[0096] Among them, different suspension control signals are generated according to the change in the centroid displacement and the change in the motion energy, so that the suspension makes different movements. In one way, the suspension is controlled to tilt according to the change in the centroid displacement. For example, when the moving object moves laterally rapidly, the suspension tilts left and right. The motion intensity of the suspension is controlled according to the change in the motion energy of the moving object. For example, violent up and down vibrations can make the suspension move up and down to simulate the vibration effect.

[0097] Second, in the moving view mode, switch the motion mode to the moving view mode, generate a suspension control signal according to the attitude information of the imaging device, and control the suspension movement according to the suspension control signal. Specifically:

[0098] 221: Analyze the essential matrix of the image to obtain the attitude information of the imaging device;

[0099] Among them, the attitude information includes at least one of the yaw angle, pitch angle, and roll angle. The suspension control device can obtain the attitude information of the imaging device through algorithms such as "Structure from Motion (SFM)".

[0100] 222: Generate a suspension control signal according to the attitude information of the imaging device, and control the suspension movement according to the suspension control signal;

[0101] Among them, the suspension control converts the attitude information of the imaging device into a suspension control signal, and controls the suspension to simulate the movement of the imaging device according to the suspension control signal.

[0102] Third, in the hybrid motion mode, switch the motion mode to the hybrid motion mode, generate a suspension control signal according to the attitude information of the imaging device and the motion intensity of the moving objects in the picture, and control the suspension movement according to the suspension control signal. Specifically:

[0103] 231: Analyze the image to obtain the global average displacement of the imaging device, and perform object detection and object tracking on the moving objects in the picture to determine the motion intensity of each moving object;

[0104] Among them, the exercise intensity includes at least one of speed, acceleration, and the change in exercise energy. It should be noted that when there are multiple moving objects in the video frame, the energy intensities of all moving objects in the video frame need to be weighted and averaged. The weighted average of other exercise intensities such as speed and acceleration refers to the weighted average of the change in exercise energy, that is, the speed or acceleration of each moving object is weighted and averaged according to the weight of each moving object, which will not be elaborated here;

[0105] 232: Control the suspension to perform translational motion according to the global average displacement of the imaging device, and control the motion intensity of the suspension according to the motion intensity of the moving object;

[0106] When controlling the suspension motion, the present application superimposes the motion influences of two types of subjects, namely the imaging device and the moving subject. When superimposing, the global average displacement of the imaging device and the motion intensity of the moving subject are mainly considered. This is because the imaging device reflects the global motion L, and the global motion L will directly affect the user's overall perception. Therefore, in this mode, the present application controls the suspension to perform translation in the X-axis, Y-axis, and Z-axis directions according to the global average displacement of the imaging device, while the moving subject reflects the local motion S. The local motion S describes the independently moving objects (such as people, vehicles, etc.) in the scene, and the local motion S will affect the user's perception of the motion intensity, such as perceiving the vibration of the moving object. Therefore, in this mode, the present application controls the motion intensity of the suspension according to the motion intensity of the moving object. For example, the suspension is controlled to move up and down according to the motion intensity of the suspension to simulate the vibration effect.

[0107] Fourth, in the dynamic background mode, switch the motion mode to the dynamic background mode, generate a suspension control signal according to at least one of the optical flow displacement amplitude, motion intensity, and motion frequency of the environment, and control the suspension motion according to the suspension control signal. Specifically:

[0108] 241: Detect the optical flow displacement amplitude, motion frequency, and / or displacement direction of the environment in the video frame;

[0109] Among them, the suspension control device can obtain the optical flow displacement amplitude of the environment by analyzing the optical flow displacement amplitude A of the background area background obtain the optical flow displacement amplitude of the environment; perform periodic motion analysis through Fourier transform to obtain the motion frequency of the environmental motion / background motion; calculate the displacement direction of the background motion through optical flow analysis technology;

[0110] After step 241, perform at least one of steps 242 to 244. When multiple steps are executed, the execution order is not limited and can be executed serially or in parallel:

[0111] 242: Determine the motion amplitude of the suspension according to the optical flow displacement amplitude of the environmental motion;

[0112] Among them, the amplitude of the optical flow displacement of the environmental motion determines the amplitude of the suspension motion. The environmental motion refers to the environmental periodic motion. If the amplitude of the optical flow displacement of the environmental periodic motion is small, it means that the intensity of the environmental periodic motion is low. At this time, the motion amplitude of the suspension motion is small, and the suspension motion can be relatively gentle, imitating slight environmental vibrations. If the amplitude of the optical flow displacement of the environmental periodic motion is large, it means that the intensity of the environmental periodic motion is high. At this time, the suspension motion can be more violent, simulating stronger external motions, such as violent changes in scenery, falling raindrops, etc.

[0113] 243: Determine the response frequency of the suspension according to the motion frequency of the environmental motion;

[0114] Among them, the response frequency of the suspension is determined according to the motion frequency of the environmental motion, so that the suspension system can accurately synchronize with the periodic changing frequency of the background motion (such as raindrop frequency, snow frequency, vibration frequency of other periodic motions, etc.), and the action frequency of the suspension can match it to provide smooth and continuous physical feedback to ensure that the cabin is synchronized with the background motion.

[0115] 244: Determine the movement direction of the suspension according to the displacement direction of the environmental movement;

[0116] The suspension is controlled to make corresponding lifting, tilting or left-right movement according to the displacement direction of the environmental movement. For example, if the background shows significant up-and-down periodic movement (such as raindrops, up-and-down shaking pictures), the suspension can make up-and-down movements to simulate the ups and downs of the car body. For another example, if there is a significant lateral periodic change in the background (such as the movement of the scenery outside the car window), the suspension can make a left-and-right tilting movement.

[0117] Fifth, in the fast switching mode, the suspension maintains a neutral position and does not transmit any control signals, thereby reducing excessive movement caused by shot editing or strong motion of the picture, reducing the user experience and improving the user's riding comfort.

[0118] Sixth, in the non-standard mode, the suspension is controlled to maintain a neutral position, no control signal is transmitted, and unnecessary movement of the suspension due to noise is minimized to improve the user's riding comfort.

[0119] It can be seen that this application can better simulate the vehicle motion feeling in the real environment through the careful analysis of the image motion characteristics and the fine control of the suspension, avoiding the unnatural or uncomfortable feedback caused by noise or imprecise control of the traditional system. This provides users with a more natural, smooth and realistic motion feeling, especially when watching movies for a long time or experiencing immersion, which can greatly reduce discomfort and improve the overall comfort of users, thus optimizing the comfort and naturalness of the user experience.

[0120] It should be noted that the screen parsing and suspension control of this solution can be real-time, without the need to pre-parse the entire video screen and preset the entire process of suspension control. Specifically, the current target frame and the next target frame of the video screen are parsed to obtain at least one subject that causes the change of the screen. Among them, the subject that causes the change of the screen includes at least one of the imaging device, the moving object in the screen, and the environment; determine the corresponding target motion mode according to at least one subject; when playing the next target frame, switch the motion mode to the target motion mode to control the suspension to move accordingly following the motion of the subject, so that the user can obtain a real-time and true motion feeling when being in the environment of the video.

[0121] The current target frame and the next target frame can be two adjacent frames in the video screen, or two non-adjacent frames separated by M frames in the video screen, where M is greater than 1. Whether they are adjacent and the number of frames separated depends on the balance between real-time requirements and computational resource optimization. The larger M is, the higher the real-time performance, the more synchronized the video screen and the suspension movement, and the greater the computational resource requirements for the hardware device. On the contrary, the smaller M is, the lower the real-time performance and the smaller the computational resource requirements. Therefore, it can be set according to specific needs, and this application does not limit it.

[0122] Preferably, M is the maximum number of frames when the computing device provides the maximum computing power. At this time, the real-time performance is the upper limit that the current computing device can achieve. Moreover, it can also receive the adjustment instruction of the user. The adjustment instruction is used to indicate the set value of M or indicate to increase or decrease the current M. When receiving the adjustment instruction of the user, adjust M according to the adjustment instruction of the user to improve the user experience. Because when there are various motions in the video screen and the motion types change very frequently, it may cause the motion mode to switch too frequently and sensitively, resulting in very unstable suspension control, thus reducing the user experience. And this method allows the user to flexibly adjust according to their own needs and the actual video screen, so the user experience can be improved and the augmented reality simulation effect can be enhanced.

[0123] The present application also provides a suspension control device. In the embodiments of the present application, the device can be divided into functional modules according to the above method examples. For example, each functional module can be corresponding to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation. Specifically, the suspension control device includes: an analysis unit 310, configured to analyze the main body causing the change of the image according to the image, and determine the motion mode corresponding to the main body, where the main body includes at least one of an imaging device, a moving object in the image, and the environment; a control unit 320, configured to switch the motion mode to the motion mode corresponding to the main body, so as to control the suspension to perform corresponding motion following the motion of the main body.

[0124] In an implementable manner, the foregoing analysis unit 310 is specifically configured to: analyze the main body causing the change of the image according to the image; if the analysis result is one of a moving object, an imaging device, and the environment, the corresponding motion modes of the main bodies are a fixed view mode, a moving view mode, and a dynamic background mode respectively; if the analysis result is an imaging device and a moving object, the corresponding motion mode is a mixed view mode; if the analysis result is no main body, the corresponding motion mode is a non-compliance mode or a quick switch mode, where the suspension is prohibited from moving in the non-compliance mode and the quick switch mode.

[0125] In an implementable manner, the foregoing analysis unit 310 is specifically configured to: detect the proportion of active pixels in the image, and the region with the maximum energy in the image; if the proportion of active pixels in the image is less than the corresponding threshold, there is a region with the maximum energy, and there is violent motion in the region with the maximum energy, then it is determined that the main body causing the change of the image is a moving object.

[0126] In an implementable manner, the foregoing analysis unit 310 is specifically configured to: detect the proportion of active pixels in the image, and the degree of consistency of the optical flow direction in the image; if the proportion of active pixels in the image is greater than the corresponding threshold, and the degree of consistency of the optical flow direction in the image is greater than the corresponding threshold, then it is determined that the main body causing the change of the image is an imaging device.

[0127] In an implementable manner, the foregoing analysis unit 310 is specifically configured to: detect the proportion of active pixels in the image, and the global motion and local motion of the image; if the proportion of active pixels in the image belongs to the corresponding preset interval, the significance degree of the global motion is greater than the corresponding threshold and the richness degree of the local motion is greater than the corresponding threshold, then it is determined that the main body causing the change of the image is an imaging device and a moving object.

[0128] In one implementable manner, the foregoing parsing unit 310 is specifically configured to: detect the degree of motion fluctuation of the background area of the image frame and the significance level of the periodic motion; if the degree of motion fluctuation of the background area of the image frame is less than the corresponding threshold and the significance level of the periodic motion of the background area is greater than the corresponding threshold, determine that the entity causing the change in the frame is the environment.

[0129] In one implementable manner, the foregoing parsing unit 310 is specifically configured to: detect the edge density or the edge density change rate, and detect the global average displacement or the proportion of active pixels; if the edge density or the edge density change rate is greater than the corresponding threshold, determine that the frame has mutated and there is no entity, where, in the case of frame mutation, no entity corresponds to the fast switching mode; if the global average displacement or the proportion of active pixels is less than the corresponding threshold, determine that the change in the frame is not significant and there is no entity, where, in the case of insignificant frame change, no entity corresponds to the non-compliance mode.

[0130] The present application further provides a suspension control device, which may include: a processor 410 and a memory 420. The above-mentioned processor 410 and memory 420 are connected through a bus 430. The processor 410 is configured to execute multiple instructions; the memory is configured to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor to perform the suspension control method as described in the above embodiments. Among them, the processor may be an Electronic Control Unit (ECU), a central processing unit (CPU), a general-purpose processor, a coprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may also be a combination for implementing computing functions, such as a combination including one or more microprocessors, a combination of 5SP and a microprocessor, and so on. In this embodiment, the processor may adopt a single-chip microcomputer, and various control functions can be realized by programming the single-chip microcomputer. The processor has the advantages of powerful computing ability and fast processing speed. Specifically, the processor 410 is configured to execute the functions of the parsing unit 310, to parse the entity causing the change in the frame based on the image frame and determine the motion mode corresponding to the entity, where the entity includes at least one of an imaging device, a moving object in the frame, and the environment; the processor 410 is further configured to execute the functions of the control unit 320, to switch the motion mode to the motion mode corresponding to the entity, so as to control the suspension to perform corresponding motion following the motion of the entity.

[0131] In one possible implementation, the foregoing processor 410 is specifically configured to: analyze the main body causing the change of the picture based on the video picture; if the analysis result is one of the main bodies of the moving object, the imaging device, and the environment, the corresponding motion modes of the respective main bodies are the fixed perspective mode, the moving perspective mode, and the dynamic background mode; if the analysis result is the imaging device and the moving object, the corresponding motion mode is the hybrid perspective mode; if the analysis result is no main body, the corresponding motion mode is the non-compliance mode or the fast switching mode, wherein the suspension is prohibited from moving in the non-compliance mode and the fast switching mode.

[0132] In one possible implementation, the foregoing processor 410 is specifically configured to: detect the proportion of active pixels in the video picture and the area with the maximum energy in the video picture; if the proportion of active pixels in the video picture is less than the corresponding threshold, there is an area with the maximum energy, and there is violent movement in the area with the maximum energy, then it is determined that the main body causing the change of the picture is a moving object.

[0133] In one possible implementation, the foregoing processor 410 is specifically configured to: detect the proportion of active pixels in the video picture and the degree of consistency of the optical flow direction in the video picture; if the proportion of active pixels in the video picture is greater than the corresponding threshold and the degree of consistency of the optical flow direction in the video picture is greater than the corresponding threshold, then it is determined that the main body causing the change of the picture is the imaging device.

[0134] In one possible implementation, the foregoing processor 410 is specifically configured to: detect the proportion of active pixels in the video picture, and the global motion and local motion of the video picture; if the proportion of active pixels in the video picture belongs to the corresponding preset interval, the degree of significance of the global motion is greater than the corresponding threshold and the degree of richness of the local motion is greater than the corresponding threshold, then it is determined that the main bodies causing the change of the picture are the imaging device and the moving object.

[0135] In one possible implementation, the foregoing processor 410 is specifically configured to: detect the degree of motion fluctuation of the background area of the video picture and the degree of significance of the periodic motion; if the degree of motion fluctuation of the background area of the video picture is less than the corresponding threshold and the degree of significance of the periodic motion of the background area is greater than the corresponding threshold, then it is determined that the main body causing the change of the picture is the environment.

[0136] In one possible implementation, the foregoing processor 410 is specifically configured to: detect the edge density or the edge density transformation rate, and detect the global average displacement or the proportion of active pixels; if the edge density or the edge density transformation rate is greater than the corresponding threshold, then it is determined that the picture has a sudden change and there is no main body, wherein in the case of a sudden change of the picture, no main body corresponds to the fast switching mode; if the global average displacement or the proportion of active pixels is less than the corresponding threshold, then it is determined that the change of the picture is not significant and there is no main body, wherein in the case of a non-significant change of the picture, no main body corresponds to the non-compliance mode.

[0137] In one embodiment, the present application further provides a computer-readable storage medium storing a plurality of instructions adapted to be loaded and executed by a processor to perform the method in any of the foregoing embodiments. A processor for executing the plurality of instructions; a memory for storing the plurality of instructions, and the instructions are loaded and executed by the processor to perform the suspension control method in the above embodiments.

[0138] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0139] The above embodiments only represent several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A suspension control method, characterized in that: The method comprises: Analyzing the subject causing the picture change based on the image picture, and determining the motion mode corresponding to the subject, wherein the subject includes at least one of the imaging device, the moving object in the picture, and the environment; The motion mode is switched to the motion mode corresponding to the main body, so as to control the suspension to follow the motion of the main body and perform corresponding motion.

2. The method according to claim 1, characterized in that The step of analyzing the subject causing the picture change based on the image picture and determining the motion mode corresponding to the subject comprises: The subject that causes the picture changes based on the image analysis; If the analysis result is a moving object, an imaging device, and a subject in the environment, the motion modes corresponding to each subject are a fixed perspective mode, a moving perspective mode, and a dynamic background mode; If the analysis result is an imaging device and a moving object, the corresponding motion mode is a mixed perspective mode; If the analysis result is that there is no main body, the corresponding movement mode is a non-standard mode or a fast switching mode, wherein the suspension is prohibited from moving in the non-standard mode and the fast switching mode.

3. The method according to claim 1 or 2, characterized in that: The steps of analyzing the main body of the image and causing the image change include: Detecting the proportion of active pixels in an image frame and the area with the highest energy in the image frame; If the proportion of active pixels in the image is less than the corresponding threshold, there is a maximum energy area, and there is violent motion in the maximum energy area, then it is determined that the subject causing the image change is a moving object.

4. The method according to claim 1 or 2, characterized in that: The steps of analyzing the main body of the image and causing the image change include: Detecting the proportion of active pixels in an image frame and the degree of consistency of the optical flow direction of the image frame; If the proportion of active pixels in the image picture is greater than the corresponding threshold, and the consistency of the optical flow direction of the image picture is greater than the corresponding threshold, it is determined that the subject causing the picture change is the imaging device.

5. The method according to claim 1 or 2, characterized in that: The steps of analyzing the main body of the image and causing the image change include: Detecting the proportion of active pixels in an image frame, and the global motion and local motion of the image frame; If the proportion of active pixels in the image frame belongs to the corresponding preset interval, the significance of global motion is greater than the corresponding threshold and the richness of local motion is greater than the corresponding threshold, it is determined that the main body causing the image change is the imaging device and the active object.

6. The method according to claim 1 or 2, characterized in that: The steps of analyzing the main body of the image and causing the image change include: Detect the degree of motion fluctuation and the significance of periodic motion in the background area of ​​the image; If the degree of motion fluctuation of the background area of ​​the image is less than the corresponding threshold and the significance of the periodic motion of the background area is greater than the corresponding threshold, it is determined that the subject causing the image change is the environment.

7. The method according to claim 1 or 2, characterized in that: The steps of analyzing the main body of the image and causing the image change include: Detect edge density or edge density change rate, and detect global average displacement or active pixel ratio; If the edge density or the edge density change rate is greater than the corresponding threshold, it is determined that the picture has a sudden change and there is no subject, wherein in the case of a sudden change in the picture, no subject corresponds to a fast switching mode; If the global average displacement or the proportion of active pixels is less than the corresponding threshold, it is determined that the picture changes are not significant and there is no subject. In the case where the picture changes are not significant, no subject corresponds to the non-standard mode.

8. A suspension control device, characterized in that: The suspension control device comprises: A parsing unit, configured to parse a subject causing the picture change based on the image picture and determine a motion mode corresponding to the subject, wherein the subject includes at least one of an imaging device, an active object in the picture, and an environment; The control unit is used to switch the motion mode to the motion mode corresponding to the main body, so as to control the suspension to follow the motion of the main body and perform corresponding motion.

9. A suspension control device, characterized in that: The suspension control device comprises a processor and a memory, wherein the memory stores a program or an instruction, and when the program or the instruction is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program or an instruction. When the program or the instruction is executed on a device, the device executes the method according to any one of claims 1 to 7.

Images