In-vehicle graffiti control method and device and medium

Abstract

The invention discloses an in-vehicle graffiti control method and device and a medium, and the method is applied to a target vehicle, and comprises the steps: obtaining a hand image collected by a binocular camera; performing gesture analysis on the hand image to determine a gesture drawing instruction; if the gesture drawing instruction is continuous drawing, generating a drawing track; rendering the drawing track into a drawing picture; the first display screen is controlled to display the drawing picture; timing is started from the continuous drawing ending moment, and whether a picture stylization instruction output by a user is collected within a preset duration or not is determined; and if yes, converting the drawing picture into a stylized picture according to the picture stylization instruction. Therefore, in-vehicle gesture drawing is carried out based on the binocular camera, drawing creation can be completed through natural gesture actions, and the entertainment experience of a user in the vehicle is improved. Besides, the user can convert the painting picture into a stylized picture according to own preferences and creativity, so that the personalization and artistry of painting creation are improved, and the aesthetic requirements of different users are met.

Description

Technical Field

[0001] The present application relates to the field of graffiti technology, and in particular to a method, device, and medium for controlling in-vehicle graffiti. Background Art

[0002] With the rapid development of smart cars, cars have gradually transformed from a means of transportation to a mobile living space that hosts many activities for users. Especially in vehicles with larger spaces, many families bring their children to play in the car, for example, providing them with entertainment experiences such as encyclopedia questions and answers.

[0003] In the context of smart car design and user experience improvement, how to add painting technology in the car to provide users with a richer, more intuitive and more interesting in-car environment experience is one of the most important research directions for technicians in this field. Summary of the Invention

[0004] In view of this, one aspect of the present application provides a method for controlling in-vehicle graffiti, which is applied to a target vehicle, and the method includes:

[0005] Obtain hand images captured by the binocular camera;

[0006] Performing gesture analysis on the hand image to determine gesture drawing instructions;

[0007] If the gesture drawing instruction is a continuous drawing, generating a drawing trajectory; rendering the drawing trajectory into a drawing image; and controlling the first display screen to display the drawing image;

[0008] Starting from the moment the continuous drawing ends, determining whether a user-entered image stylization instruction is collected within a preset time period;

[0009] If so, the painting image is converted into a stylized image according to the image stylization instruction.

[0010] Optionally, performing gesture analysis on the hand image to determine the gesture instruction type includes:

[0011] constructing a three-dimensional hand image sequence according to the hand image;

[0012] Extracting gesture features from the three-dimensional hand image sequence to obtain a two-dimensional feature map sequence; the gesture features include geometric features, dynamic features and spatial features;

[0013] Based on a pre-constructed mapping relationship between gestures and instructions, gesture analysis is performed on the two-dimensional feature image sequence to determine the gesture instruction type.

[0014] Optionally, the hand image includes a first image captured by an RGBD camera and a second image captured by a TOF camera; and constructing a three-dimensional hand image sequence based on the hand image includes:

[0015] Acquire color information and first depth information of the first image, and second depth information of the second image;

[0016] marking common feature points of the first image and the second image to obtain a feature point marking map;

[0017] Based on the triangulation principle, the feature point annotation image is calculated to obtain third depth information;

[0018] The three-dimensional image sequence of the hand is constructed based on the color information, the first depth information, the second depth information, and the third depth information; wherein the first depth information, the second depth information, and the third depth information are all distance information between the hand and the binocular camera.

[0019] Optionally, extracting gesture features from the three-dimensional hand image sequence includes:

[0020] fusing the first depth information, the second depth information, and the third depth information into target depth information;

[0021] Preprocessing the three-dimensional hand image sequence; and segmenting the hand contour from the preprocessed three-dimensional hand image sequence;

[0022] extracting contour features and finger features from the hand contour according to the color information and the target depth information to obtain the geometric features;

[0023] Based on the target depth information, the dynamic features are determined by a video processing algorithm, and the spatial features are extracted by a convolutional neural network.

[0024] Optionally, converting the painting image into a stylized image according to the image stylization instruction includes:

[0025] Extracting keywords from the image stylization instruction;

[0026] By specifying a large language model, analyzing the keywords and the painting image to determine a target conversion style;

[0027] Based on the target conversion style, the painting image is converted into the stylized image through a diffusion model.

[0028] Optionally, the in-vehicle graffiti control method further includes:

[0029] When a multi-screen drawing instruction is obtained, controlling the first display screen and the second display screen to be synchronously connected;

[0030] Obtaining a vehicle-mounted drawing instruction collected when the second display screen is triggered;

[0031] generating the painting image and the stylized image according to the vehicle-mounted painting instruction and the gesture painting instruction;

[0032] The first display screen and the second display screen are controlled to synchronously display the painting image and the stylized image.

[0033] Optionally, generating a drawing trajectory includes:

[0034] Recognizing the hand image to determine the finger brush;

[0035] Obtaining the initial position coordinates of the finger brush in the hand image at different moments;

[0036] After filtering the initial position coordinates, the target position coordinates are obtained;

[0037] Fitting the target position coordinates based on the movement information of the finger brush to obtain an initial trajectory; wherein the movement information includes movement speed and movement direction;

[0038] The initial trajectory is optimized to obtain the painting trajectory.

[0039] Another aspect of the present application provides a device for controlling graffiti in a vehicle, the device comprising:

[0040] A hand image acquisition module is used to acquire hand images captured by a binocular camera;

[0041] A hand image analysis module, configured to perform gesture analysis on the hand image to determine a gesture drawing instruction;

[0042] a drawing picture generating module, configured to generate a drawing trajectory if the gesture drawing instruction is a continuous drawing; render the drawing trajectory into a drawing picture; and control the first display screen to display the drawing picture;

[0043] The stylized image conversion module is used to start timing from the moment the continuous painting ends and determine whether a picture stylization instruction input by the user is collected within a preset time period; if so, convert the painting picture into a stylized picture according to the picture stylization instruction.

[0044] Another aspect of the present application provides an in-vehicle graffiti control device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the in-vehicle graffiti control method when executing the program.

[0045] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the in-vehicle graffiti control method when executed by a processor.

[0046] The present application provides a method, device, and medium for controlling in-vehicle graffiti. The beneficial effects include: improved gesture recognition accuracy through in-vehicle gesture painting using a binocular camera. Users can create paintings through natural hand gestures without the need for handheld drawing tools, resulting in convenient operation and significantly enhanced in-vehicle entertainment and interactive fun. Furthermore, users can transform their paintings into stylized images based on their preferences and creativity, increasing the personalization and artistry of their paintings and satisfying the aesthetic needs of diverse users. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of a method for controlling in-vehicle graffiti provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram showing the principle of a method for controlling graffiti in a vehicle provided by an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the effects of a painting picture and a stylized picture provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram showing the principle of a method for controlling graffiti in a vehicle provided by another embodiment of the present application;

[0051] Figure 5 A schematic diagram showing the principle of a multi-screen graffiti control method provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a vehicle graffiti control device provided in an embodiment of the present application;

[0053] Figure 7 This is a schematic structural diagram of an in-vehicle graffiti control device provided by another embodiment of the present application.

[0054] The accompanying drawings are marked as follows: 60 is a hand image acquisition module, 61 is a hand image analysis module, 62 is a painting picture generation module, 63 is a stylized picture conversion module, 70 is a memory, 71 is a processor, 72 is a display screen, 73 is an input and output interface, 74 is a communication interface, 75 is a power supply, 76 is a communication bus, 701 is a computer program, 702 is an operating system, and 703 is data. DETAILED DESCRIPTION

[0055] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0056] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0057] Figure 1 This is a flow chart of a method for controlling graffiti in a vehicle provided by an embodiment of the present application, as shown in FIG. Figure 1 As shown, the method includes:

[0058] S10: Acquire a hand image captured by a binocular camera;

[0059] First of all, it should be noted that the control method provided in the embodiments of the present application is applied to a target vehicle, and the target vehicle may include but is not limited to a sedan, a sport utility vehicle (SUV), a multi-purpose vehicle (MPV), an off-road vehicle, a pickup truck, or other power-driven non-track-borne vehicle.

[0060] In a specific embodiment, a user can enter the drawing interface by clicking on the display or using voice commands. Furthermore, to accurately obtain information about the hand drawing in the air, a binocular camera is installed in the target vehicle. The binocular camera collects hand images, including the drawing hand, in real time, providing a high-quality hand image sequence for subsequent gesture analysis. It should be noted that this application does not limit the installation location of the binocular camera.

[0061] S11: performing gesture analysis on the hand image to determine gesture drawing instructions;

[0062] S12: If the gesture drawing instruction is a continuous drawing, generating a drawing trajectory; rendering the drawing trajectory into a drawing image; and controlling the first display screen to display the drawing image;

[0063] Furthermore, gesture analysis is performed on the hand image captured by the binocular camera to determine the user's drawing intent. Gesture drawing instructions include, but are not limited to, continuous drawing, clearing the canvas, and switching breakpoints. In an optional embodiment, when the hand image analysis results indicate that the gesture drawing instruction is continuous drawing, a drawing trajectory is generated.

[0064] Specifically, hand image analysis can identify the user's drawing finger, i.e., the fingerbrush. Furthermore, the hand image can be used to obtain the fingerbrush's position information. The fingerbrush is then mapped to a focal point on the in-car display screen. Based on the position information generated by the fingerbrush's movement at different times, the drawing trajectory is derived.

[0065] Furthermore, the drawing trajectory is rendered as a drawing image, and the first display screen is controlled to display the drawing image in real time. It should be noted that multiple display screens can be provided in the target vehicle, and this application does not limit the first display screen that displays the drawing image. In an optional embodiment, multiple display screens can be synchronously connected to display the drawing image simultaneously.

[0066] In an optional embodiment, the hand image is analyzed. If it is determined that the user is performing a tapping motion, the gesture drawing instruction can be determined as a switch breakpoint. At the breakpoint, the user can move the hand without generating a trace. When the user selects to continue drawing on the screen, the drawing resumes. In other words, when the drawing resume instruction is obtained, the continuous drawing mode is entered again.

[0067] When the user puts down the finger brush, the painting is paused, and when the finger is raised again, the painting mode is triggered again and the user can continue painting. In another optional embodiment, moving the palm left and right can clear the canvas, and moving the palm up and down can view historical painting pictures.

[0068] Of course, in an optional embodiment, the user can use specific finger movements to select parameters such as the type of drawing tool (e.g., brush, pencil, eraser, etc.), drawing color, and drawing line thickness, thereby achieving precise control over the drawing process. It should be noted that the gesture actions corresponding to different gesture drawing instructions are not limited in this application.

[0069] S13: Starting from the moment the continuous drawing ends, determining whether the image stylization instruction input by the user is collected within a preset time period;

[0070] S14: If yes, convert the painting image into a stylized image according to the image stylization instruction.

[0071] After the painting is completed, the user can end the current painting through a preset gesture. At this time, the user can output a picture stylization instruction, where the picture stylization instruction can be voice or text. This application does not limit the form of the picture stylization instruction.

[0072] If, starting from the moment the continuous drawing process ends, a user input image stylization instruction is received within a preset time period, such as 15 seconds, the image stylization instruction can be parsed to understand the user's intention for the final generated image. The stylized image can then be combined with the image stylization instruction to convert the drawing image into a stylized image. Stylized images may include, but are not limited to, cartoon-style images, oil painting-style images, and watercolor-style images.

[0073] Of course, if the image stylization instruction output by the user is not obtained within the preset time, the painting image is directly analyzed to determine the painting content included in the image, that is, the painting image is understood, and the painting image is further converted into a stylized image based on the understanding result.

[0074] In an optional embodiment, a diffusion model may be used for conversion to a stylized image. A large language model may be used for recognition and understanding of stylized image instructions and painted images, which is not limited in this application.

[0075] Figure 2 This is a schematic diagram of the principle of a method for controlling graffiti in a vehicle provided by an embodiment of the present application, as shown in FIG. Figure 2 As shown, the target vehicle includes a central control entertainment host, a binocular camera, and a display screen, wherein the display screen includes a first display screen and a second display screen, and the central control entertainment host includes an image processing module and a picture conversion module. In a specific embodiment, the first display screen and the second display screen can be drawn synchronously.

[0076] In a specific embodiment, after a user enters the drawing interface, a binocular camera captures a hand image in real time and transmits it to the central control entertainment console. The image processing module in the central control entertainment console then performs gesture analysis on the hand image, generates a drawing trajectory, and renders the drawing trajectory to produce a drawing image. After processing, the drawing image is displayed in real time on the first and / or second display screens.

[0077] Furthermore, the generated painting image is transmitted to the image conversion module so that the image conversion module converts the painting image into a stylized image. Similarly, after the conversion, the stylized image is displayed on the first display screen and / or the second display screen.

[0078] Figure 3 This is a schematic diagram of the effects of a painting picture and a stylized picture provided in an embodiment of the present application, such as Figure 3 As shown, the left side is a painting image generated according to the user's painting trajectory, and the right side is a stylized image after the painting image is converted.

[0079] Thus, the in-car graffiti control method provided by the present embodiment uses binocular cameras to perform in-car gesture painting, improving gesture recognition accuracy. Users can create paintings through natural hand gestures without holding any drawing tools. This convenient operation greatly enhances the user's in-car entertainment experience and interactive fun. Furthermore, users can transform their paintings into stylized images based on their preferences and creativity, increasing the personalization and artistry of their paintings and satisfying the aesthetic needs of different users.

[0080] In an optional embodiment, performing gesture analysis on the hand image to determine the gesture instruction type includes:

[0081] Constructing a three-dimensional hand image sequence according to the hand image;

[0082] Extract gesture features from a three-dimensional hand image sequence to obtain a two-dimensional feature map sequence; gesture features include geometric features, dynamic features, and spatial features;

[0083] Based on the pre-built mapping relationship between gestures and commands, gesture analysis is performed on the two-dimensional feature map sequence to determine the gesture command type.

[0084] It is understood that in this specific embodiment, the hand image captured by the binocular camera is a two-dimensional image. To accurately identify the type of gesture command, the two-dimensional image needs to be converted into a three-dimensional image. In an optional embodiment, before converting the two-dimensional image into a three-dimensional image, in order to improve the accuracy of gesture command type recognition, the hand image can be pre-processed. The pre-processing may include but is not limited to removing noise, adjusting brightness, and adjusting contrast.

[0085] Figure 4 This is a schematic diagram of the principle of a method for controlling graffiti in a vehicle provided by another embodiment of the present application, as shown in FIG. Figure 4As shown, a three-dimensional hand image sequence is constructed based on the hand image captured by the binocular camera. Specifically, in an optional embodiment, the hand area can be located based on the depth information in the hand image. Further, the hand image in the depth image can be segmented based on the depth threshold. For example, a threshold is set (for example, 10 cm), and the hand image whose distance from the binocular camera is within the threshold range is extracted, and the hand image is three-dimensionally projected to obtain a three-dimensional image. It can be understood that in the specific embodiment of painting, the binocular camera captures hand images in real time. Therefore, the three-dimensional image is time-serialized to form a three-dimensional hand image sequence, thereby ensuring the continuity of the hand movement and providing a basis for the subsequent gesture instruction type recognition.

[0086] Furthermore, after obtaining the three-dimensional image sequence of the hand, as Figure 4 As shown, gesture features are extracted from a 3D hand image sequence to produce a 2D feature map sequence. Gesture features include geometric, dynamic, and spatial features. Geometric features are static features that may include, but are not limited to, information such as the shape, size, and position of the hand. Dynamic features reflect changes in hand movements, including, but not limited to, trajectory, velocity, and acceleration. Spatial features describe the position and posture of the hand in 3D space.

[0087] The extracted geometric features, dynamic features and spatial features are fused to generate Figure 4 The two-dimensional feature graph sequence shown contains comprehensive information about the gesture, providing rich data support for subsequent gesture analysis. Specifically, in an optional embodiment, the two-dimensional feature graph sequence is analyzed based on a pre-established mapping relationship between gestures and commands to determine the gesture command type.

[0088] This application does not specifically limit the mapping between gestures and commands. Gestures include, but are not limited to, spreading five fingers, tapping, sliding your palm left and right, sliding your palm up and down, and the OK gesture. Commands include, but are not limited to, switching breakpoints, continuous drawing, clearing the canvas, selecting a brush, selecting a drawing color, selecting a drawing line, adjusting a drawing line, pausing drawing, continuing drawing, starting drawing, and ending drawing.

[0089] In this way, efficient conversion from hand images to gesture commands is achieved, providing technical support for in-car graffiti, meeting the diverse needs of users and enhancing the fun.

[0090] Based on the above embodiment, as an optional embodiment, Figure 4As shown, the binocular camera includes an RGBD camera and a TOF camera. Correspondingly, the captured hand image includes a first image captured by the RGBD camera and a second image captured by the TOF camera. Therefore, in an optional embodiment, constructing a three-dimensional hand image sequence based on the hand image includes:

[0091] Acquire color information and first depth information of the first image, and second depth information of the second image;

[0092] Marking the same feature points of the first image and the second image to obtain a feature point marking map;

[0093] Based on the triangulation principle, the feature point annotation map is calculated to obtain the third depth information;

[0094] A three-dimensional image sequence of the hand is constructed according to the color information, the first depth information, the second depth information, and the third depth information; wherein the first depth information, the second depth information, and the third depth information are all distance information between the hand and the binocular camera.

[0095] In a specific embodiment, the visual side combines an RGBD camera and a TOF camera, wherein the first image captured by the RGBD camera can simultaneously provide color information (RGB) and depth information (Depth), that is, the first depth information. The depth image provides information about the distance between the hand and the camera. The color information can be used to capture the color features of the hand, facilitating subsequent gesture recognition and segmentation. The first depth information provides depth information of the hand and can be used to construct a three-dimensional image sequence of the hand.

[0096] The second image captured by the TOF (Time-of-Flight) camera can provide second depth information. In a specific embodiment, the TOF camera uses a multi-point emission and measurement method to calculate the second depth information. Specifically, the time-of-flight method is used to continuously emit light pulses (generally invisible light) to the object being measured, and then receive the light pulses reflected from the object. The distance of the object being measured from the camera is calculated by detecting the flight (round-trip) time of the light pulses. The TOF camera can make up for the shortcomings of the RGBD camera in low-light conditions and provide more accurate depth information.

[0097] Furthermore, in an optional embodiment, a binocular camera captures two different planar images, namely, a first image and a second image. Common feature points in the first and second images are annotated, for example, key points of the hand, such as the wrist and finger joints, to generate a feature point annotation map. Third depth information is then calculated from the feature point annotation map based on the principle of triangulation. The triangulation principle calculates the depth of an object using the parallax between the two cameras, thereby providing more accurate depth information.

[0098] In an optional embodiment, the first depth information, the second depth information, and the third depth information are fused to obtain accurate target depth information. This application does not limit the fusion method. Furthermore, a three-dimensional image sequence of the hand is constructed based on the color information and the target depth information. Specifically, for each pixel point, its coordinates in three-dimensional space are calculated by combining its color information and target depth information, and the three-dimensional coordinates of all pixels are combined into point cloud data to obtain a three-dimensional image. The three-dimensional image is time-series processed to obtain a three-dimensional image sequence of the hand.

[0099] Based on the above embodiment, as an optional embodiment, extracting gesture features from a three-dimensional hand image sequence includes:

[0100] fusing the first depth information, the second depth information, and the third depth information into target depth information;

[0101] Preprocessing a three-dimensional hand image sequence; and segmenting a hand contour from the preprocessed three-dimensional hand image sequence;

[0102] According to the color information and target depth information, the contour features and finger features are extracted from the hand contour to obtain the geometric features;

[0103] Based on the target depth information, dynamic features are determined through video processing algorithms, and spatial features are extracted through convolutional neural networks.

[0104] In a specific embodiment, the first depth information, the second depth information, and the third depth information determined based on the first image and the second image are fused into target depth information. Simultaneously, the acquired three-dimensional hand image sequence is preprocessed. Specifically, the preprocessing may include, but is not limited to, denoising, normalization, and background removal.

[0105] It is understandable that since the image acquisition process may be affected by various factors, resulting in the presence of noise, denoising is required. Furthermore, to improve the stability and accuracy of feature extraction, the image size is resized to a uniform size for subsequent processing. Simultaneously, methods such as background subtraction and a mixed Gaussian model are used to separate the hand from the background for more accurate hand feature extraction.

[0106] Furthermore, the hand contour is segmented from the pre-processed three-dimensional hand image sequence. In a specific embodiment, the geometric features are used to describe the static shape of the hand, and include contour features and finger features, wherein the contour features include but are not limited to the center of mass, perimeter, area, center point and radius of the circumscribed circle, and position of the rectangular frame of the contour. Finger features include but are not limited to the number, length, and width of fingers. In an optional embodiment, based on the acquired color information and target depth information, the geometric features of each frame of the image can be extracted by a convolutional neural network (CNN).

[0107] Dynamic features reflect the changes in hand movements, that is, the continuous changes in the hands. In fact, dynamic features can also be understood as motion features, such as the hand's motion trajectory, direction, speed, dynamic changes in gestures (for example, the direction of waving, the speed of making a fist, etc.). By analyzing the hand's motion trajectory, speed, direction, acceleration and other information in the time series, the dynamic behavior of the gesture can be identified. In an optional embodiment, the distance between the hand and the binocular camera can be determined based on the target depth information. Furthermore, the dynamic features can be determined by a video processing algorithm. For example, a recurrent neural network (RNN) or a long short-term memory network (LSTM) can be used to process a three-dimensional image sequence of the hand to extract the 3D shape features of the hand (for example, palm curvature and finger curvature, etc.) to capture the time dependence of the hand movement.

[0108] Spatial features describe the position and posture of the hand in three-dimensional space. In a specific embodiment, the spatial posture of the hand can be determined by calculating the three-dimensional coordinates of the hand joints and their mutual relationship. In an optional embodiment, the spatial features can be determined by a 3D convolutional neural network (3D CNN) or a spatiotemporal convolutional network (STCNN). Specifically, the 3D convolutional neural network can extract spatial and temporal information from the sequence at the same time, and is composed of multiple 3D convolutional layers, 3D pooling layers and activation functions. Compared with 2D, multiple feature maps can be operated on in width, height and time dimensions at the same time, so that spatial information can be captured from the sequence at the same time.

[0109] Specifically, after inputting a 3D hand image sequence, the spatial features of the image are extracted through the convolutional and pooling layers of the 3D convolutional neural network. First, video keyframes are extracted, and gesture joint information is obtained using the Openpose framework. Then, based on the dynamic gesture keyframes and gesture joint information, the angle, distance, and trajectory features of the gesture are calculated. Finally, the three types of features, angle, distance, and trajectory, are fused, and the LinerSVM model is used to implement dynamic gesture recognition. The feature map is then flattened and transferred to the fully connected layer in equal directions to complete the model's prediction, converting the 3D hand image into a 2D feature map.

[0110] In an optional embodiment, converting a painting image into a stylized image according to the image stylization instruction includes:

[0111] Extract keywords from image stylization instructions;

[0112] By specifying a large language model, keywords and painting images are analyzed to determine the target conversion style;

[0113] Based on the target conversion style, the painting image is converted into a stylized image through the diffusion model.

[0114] like Figure 4 As shown, in a specific embodiment, when the gesture instruction type is continuous drawing, a corresponding drawing trajectory is generated and rendered as a drawing image. Furthermore, within a preset time period, the user input image stylization instruction is obtained, thereby combining the drawing image and the stylization instruction to generate a stylized image.

[0115] Specifically, keywords are extracted from the image stylization instruction. For example, if the image stylization instruction is "transform this painting into a Van Gogh-style oil painting," the corresponding keywords extracted include "Van Gogh style" and "oil painting." It should be noted that this application does not limit the method for extracting keywords.

[0116] Furthermore, the large language model analyzes the keywords and the painting to determine the target conversion style. As can be appreciated, by combining the keyword content and the content displayed in the painting, the painting can be transformed into a stylized image that better reflects the user's intent. Based on the target conversion style, the painting is further transformed into a stylized image using a diffusion model. In an optional embodiment, an appropriate diffusion model can be selected based on the target style.

[0117] like Figure 4 As shown, in a specific embodiment, the generated drawing image and stylized image are displayed in real time on the first display screen, allowing the user to promptly check the current drawing status. Thus, based on the diffusion model, the trajectory of the stick figure drawn in the air is converted into an image with a stylized filter, which can compensate for the lack of completeness of the hand gesture stick figure drawing, eliminate the need for detailed drawing, and highlight the interactive effect.

[0118] Figure 5 This is a schematic diagram of the principle of a multi-screen graffiti control method provided in an embodiment of the present application. As an optional embodiment, the in-vehicle graffiti control method provided in the present application further includes:

[0119] When a multi-screen drawing instruction is obtained, the first display screen and the second display screen are controlled to be connected synchronously;

[0120] Obtaining the vehicle-mounted drawing instructions collected when the second display screen is triggered;

[0121] Generate painting images and stylized images based on vehicle-mounted painting instructions and gesture painting instructions;

[0122] The first display screen and the second display screen are controlled to synchronously display the painting image and the stylized image.

[0123] To enhance the diversity of in-car painting and meet the diverse needs of users, multiple displays can be used for painting. Specifically, upon receiving a multi-screen painting command from the user, the first and second displays are controlled to synchronize and connect. The multi-screen painting command can be input through the first or second display, or by voice input. This application does not limit the form of the multi-screen painting command.

[0124] In an optional embodiment, the first display screen is used for air drawing, and the second display screen is used for touch-based drawing. It is understood that air drawing requires a binocular camera. Considering cost, a single display screen can be used for air drawing. Since children in the rear row typically have a higher demand for drawing, the rear display screen is set as the first display screen. In an optional embodiment, the binocular camera can be installed on the ceiling of the rear seats, directly above the center of the first display screen. The second display screen can be the display screen in front of the driver's seat.

[0125] like Figure 5 As shown, when a user wants to "draw together" with their children in the front and back seats, that is, when a multi-screen drawing instruction is received, the first and second displays are synchronously connected. Specifically, the screen is projected to the other display by the user, and the images on the front and back displays are displayed synchronously in real time. At this time, the binocular camera captures the hand image to obtain the gesture drawing instruction. At the same time, the user can also use the second display to perform touch drawing to receive the drawing instruction from the vehicle computer.

[0126] Based on the vehicle-mounted drawing instructions and gesture drawing instructions, a drawing trajectory is generated and rendered into a drawing image. After obtaining the drawing image, the user's input image stylization instructions are similarly obtained and the generated drawing image is converted into a stylized image according to the image stylization instructions.

[0127] like Figure 5 As shown, in a specific embodiment, the generated painting image and stylized image are displayed synchronously and in real time on the first display screen and the second display screen. Thus, multi-screen joint painting is achieved, providing users with a rich and diverse painting experience.

[0128] In an optional embodiment, generating a drawing trajectory includes:

[0129] Recognize hand images to identify finger brushes;

[0130] Get the initial position coordinates of the finger brush in the hand image at different times;

[0131] After filtering the initial position coordinates, the target position coordinates are obtained;

[0132] Based on the movement information of the finger brush, the target position coordinates are fitted to obtain the initial trajectory; wherein the movement information includes the movement speed and movement direction;

[0133] The initial trajectory is optimized to obtain the painting trajectory.

[0134] In a specific embodiment, after the hand image is acquired by the binocular camera, the hand image is recognized to determine the finger brush. The binocular camera detects that the index finger pointing static gesture exceeds a preset time (for example, 3 seconds), and the brush cursor is displayed on the display screen as the default position.

[0135] When determining the finger brush, in an optional embodiment, the key feature points of the hand in the hand image can be first found through the feature point detection algorithm in the computer vision library (such as Harris corner detection, Shi-Tomasi corner detection, etc.). These feature points are usually distributed in the finger joints and other parts. Then, by tracking the position changes of these feature points in different frame images, the finger is determined based on their relative position relationship and the shape of the overall hand contour. For example, the position feature of the joint connecting the thumb and the palm is obvious. Its relative coordinates in the hand contour and the distance, angle and other relationships with other finger feature points can be used as a basis for distinguishing it from other fingers. The direction, length and other features of the lines connecting the feature points of each joint of the index finger, middle finger and other fingers are also different, and identification is performed by analyzing these.

[0136] In another optional embodiment, recognition can be performed using a deep learning model such as a convolutional neural network (CNN). Specifically, first, a large amount of hand image data containing different finger postures is collected and annotated (clarifying the corresponding finger state in each image). Then, this annotated data is used to train a network model, allowing the network model to learn the appearance characteristics of different fingers, image patterns under different finger movements, etc. In a specific embodiment, the trained network model can be directly called to recognize hand images to determine finger brushes.

[0137] Furthermore, the real-time trajectory of the fingerbrush can be tracked based on the initial position coordinates of the fingerbrush in the hand image at different times. In a specific embodiment, to improve drawing accuracy, after obtaining the initial position coordinates, they are filtered to remove abnormal data caused by hand shaking or image noise.

[0138] In one optional embodiment, the discrete target position coordinates at different moments are connected in chronological order to form an initial trajectory. In another optional embodiment, a smoother curve fitting method (e.g., spline curve fitting) can be used to fit the target position coordinates based on the finger brush movement information, generating a more natural, smooth, and realistic initial trajectory for the gesture movement. The movement information includes movement speed and direction.

[0139] To further improve drawing accuracy, the initial trajectory is optimized. Specifically, the impact of noise and outliers on trajectory quality is removed. During filtering, the trajectory data can be processed using filtering algorithms (such as median filtering and mean filtering) to filter out unreasonable coordinate points caused by sensor jitter, short-term misjudgments, etc. At the same time, the trajectory can be rationally judged and adjusted based on geometric relationships such as distance and angle between adjacent trajectory points, as well as time intervals, to ensure that the trajectory conforms to the physical laws and logic of the actual brush movement.

[0140] In an optional embodiment, the generation of a drawing trajectory, i.e., tracking of the trajectory, can be divided into three categories: keypoint-based tracking, region-based tracking, and deep learning-based tracking. Keypoint-based tracking detects hand keypoints (e.g., fingertips, palm center, etc.) in each frame of the hand image and tracks the movement of these keypoints using optical flow or feature matching.

[0141] For area tracking, in an optional embodiment, an object tracking algorithm (e.g., KCF, SORT, DeepSORT, etc.) can be used to track the hand area. Deep learning-based tracking can use a deep learning model to directly predict the hand position, allowing for accurate tracking of the hand even when it moves rapidly.

[0142] In the above embodiments, the in-vehicle graffiti control method is described in detail. The present application also provides a corresponding embodiment of an in-vehicle graffiti control device.

[0143] Figure 6 This is a structural diagram of a vehicle graffiti control device provided by an embodiment of the present application, such as Figure 6 As shown, the device includes:

[0144] A hand image acquisition module 60 is used to acquire a hand image captured by a binocular camera;

[0145] A hand image analysis module 61 is used to perform gesture analysis on the hand image to determine a gesture drawing instruction;

[0146] The drawing picture generating module 62 is configured to generate a drawing trajectory if the gesture drawing instruction is a continuous drawing; render the drawing trajectory into a drawing picture; and control the first display screen to display the drawing picture;

[0147] The stylized image conversion module 63 is used to start timing from the moment when the continuous painting ends and determine whether the image stylization instruction input by the user is collected within a preset time period; if so, convert the painting image into a stylized image according to the image stylization instruction.

[0148] In addition, the in-vehicle graffiti control device provided in the embodiment of the present application further includes:

[0149] A three-dimensional image sequence construction module, used for constructing a three-dimensional hand image sequence based on the hand image;

[0150] The gesture feature extraction module is used to extract gesture features from the three-dimensional hand image sequence to obtain a two-dimensional feature map sequence; gesture features include geometric features, dynamic features and spatial features;

[0151] The gesture analysis module is used to perform gesture analysis on the two-dimensional feature map sequence based on the pre-built mapping relationship between gestures and commands to determine the gesture command type.

[0152] An information acquisition module, configured to acquire color information and first depth information of a first image, and second depth information of a second image;

[0153] A feature point annotation module is used to annotate the same feature points in the first image and the second image to obtain a feature point annotation map;

[0154] A feature point annotation map calculation module is used to calculate the feature point annotation map based on the triangulation principle to obtain the third depth information;

[0155] The three-dimensional image sequence construction submodule is used to construct a three-dimensional image sequence of the hand based on color information, first depth information, second depth information and third depth information; wherein the first depth information, the second depth information and the third depth information are all distance information between the hand and the binocular camera.

[0156] a depth information fusion module, configured to fuse the first depth information, the second depth information, and the third depth information into target depth information;

[0157] A preprocessing module, configured to preprocess the three-dimensional hand image sequence; and segment the hand contour from the preprocessed three-dimensional hand image sequence;

[0158] a first feature extraction module, configured to extract contour features and finger features from the hand contour according to the color information and the target depth information, to obtain the geometric features;

[0159] The second feature extraction module is used to determine the dynamic features through a video processing algorithm based on the target depth information, and extract the spatial features through a convolutional neural network.

[0160] Keyword extraction module, used to extract keywords from image stylization instructions;

[0161] The target conversion style determination module is used to analyze keywords and painting images by specifying a large language model to determine the target conversion style;

[0162] The stylized image conversion module is also used to convert the style based on the target and convert the painting image into a stylized image through the diffusion model.

[0163] A multi-screen drawing instruction acquisition module is used to control the first display screen and the second display screen to be synchronously connected when a multi-screen drawing instruction is acquired;

[0164] The vehicle computer drawing instruction acquisition module is used to acquire the vehicle computer drawing instruction collected when the second display screen is triggered;

[0165] The target generation module is used to generate painting images and stylized images based on the vehicle-mounted painting instructions and gesture painting instructions;

[0166] The synchronization control module is used to control the first display screen and the second display screen to synchronously display the painting image and the stylized image.

[0167] A finger brush identification module, used to identify the hand image to determine the finger brush;

[0168] An initial position coordinate acquisition module is used to obtain the initial position coordinates of the finger brush in the hand image at different moments;

[0169] A filtering module is used to filter the initial position coordinates to obtain the target position coordinates;

[0170] A fitting module is used to fit the target position coordinates based on the movement information of the finger brush to obtain an initial trajectory; wherein the movement information includes movement speed and movement direction;

[0171] The optimization module is used to optimize the initial trajectory to obtain the painting trajectory.

[0172] Figure 7 This is a structural diagram of a vehicle graffiti control device provided by another embodiment of the present application, such as Figure 7 As shown, the in-vehicle graffiti control device includes: a memory 70 for storing computer programs;

[0173] The processor 71 is configured to implement the steps of the in-vehicle graffiti control method mentioned in the above embodiment when executing the computer program.

[0174] The in-vehicle graffiti control device provided in this embodiment may include, but is not limited to, a vehicle controller and a domain controller.

[0175] Among them, the processor 71 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 71 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 71 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 71 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 71 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0176] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In this embodiment, the memory 70 is used to store at least the following computer program 701. When loaded and executed by the processor 71, this computer program is capable of implementing the relevant steps of the in-vehicle graffiti control method disclosed in any of the aforementioned embodiments. Furthermore, the resources stored in the memory 70 may include an operating system 702 and data 703, which may be stored in either a temporary or permanent manner. The operating system 702 may include Windows, Unix, Linux, and the like. The data 703 may include, but is not limited to, relevant data involved in the in-vehicle graffiti control method.

[0177] In some embodiments, the in-vehicle graffiti control device may further include a display screen 72 , an input / output interface 73 , a communication interface 74 , a power supply 75 , and a communication bus 76 .

[0178] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation of the in-vehicle graffiti control device, and may include more or fewer components than shown in the figure.

[0179] The in-vehicle graffiti control device provided in an embodiment of the present application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the in-vehicle graffiti control method of the above embodiment.

[0180] It should be noted that although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

Claims

1. A method for controlling graffiti in a vehicle, characterized in that: Applied to a target vehicle, the method comprises: Obtain hand images captured by the binocular camera; Performing gesture analysis on the hand image to determine gesture drawing instructions; If the gesture drawing instruction is a continuous drawing, generating a drawing trajectory; rendering the drawing trajectory into a drawing image; and controlling the first display screen to display the drawing image; Starting from the moment the continuous drawing ends, determining whether a user-entered image stylization instruction is collected within a preset time period; If so, the painting image is converted into a stylized image according to the image stylization instruction.

2. The in-vehicle graffiti control method according to claim 1, characterized in that: The performing gesture analysis on the hand image to determine the gesture instruction type includes: constructing a three-dimensional hand image sequence according to the hand image; Extracting gesture features from the three-dimensional hand image sequence to obtain a two-dimensional feature map sequence; the gesture features include geometric features, dynamic features and spatial features; Based on a pre-constructed mapping relationship between gestures and instructions, gesture analysis is performed on the two-dimensional feature image sequence to determine the gesture instruction type.

3. The in-vehicle graffiti control method according to claim 2, characterized in that: The hand image includes a first image captured by an RGBD camera and a second image captured by a TOF camera; and constructing a three-dimensional hand image sequence based on the hand image includes: Acquire color information and first depth information of the first image, and second depth information of the second image; marking common feature points of the first image and the second image to obtain a feature point marking map; Based on the triangulation principle, the feature point annotation image is calculated to obtain third depth information; The three-dimensional image sequence of the hand is constructed based on the color information, the first depth information, the second depth information, and the third depth information; wherein the first depth information, the second depth information, and the third depth information are all distance information between the hand and the binocular camera.

4. The in-vehicle graffiti control method according to claim 3, wherein: Extracting gesture features from the three-dimensional hand image sequence includes: fusing the first depth information, the second depth information, and the third depth information into target depth information; Preprocessing the three-dimensional hand image sequence; and segmenting the hand contour from the preprocessed three-dimensional hand image sequence; extracting contour features and finger features from the hand contour according to the color information and the target depth information to obtain the geometric features; Based on the target depth information, the dynamic features are determined by a video processing algorithm, and the spatial features are extracted by a convolutional neural network.

5. The in-vehicle graffiti control method according to claim 1, wherein: The step of converting the painting image into a stylized image according to the image stylization instruction includes: Extracting keywords from the image stylization instruction; By specifying a large language model, analyzing the keywords and the painting image to determine a target conversion style; Based on the target conversion style, the painting image is converted into the stylized image through a diffusion model.

6. The in-vehicle graffiti control method according to claim 1, wherein: The method further comprises: When a multi-screen drawing instruction is obtained, controlling the first display screen and the second display screen to be synchronously connected; Obtaining a vehicle-mounted drawing instruction collected when the second display screen is triggered; generating the painting image and the stylized image according to the vehicle-mounted painting instruction and the gesture painting instruction; The first display screen and the second display screen are controlled to synchronously display the painting image and the stylized image.

7. The in-vehicle graffiti control method according to claim 1, wherein: Generating a painting trajectory includes: Recognizing the hand image to determine the finger brush; Obtaining the initial position coordinates of the finger brush in the hand image at different moments; After filtering the initial position coordinates, the target position coordinates are obtained; Fitting the target position coordinates based on the movement information of the finger brush to obtain an initial trajectory; wherein the movement information includes movement speed and movement direction; The initial trajectory is optimized to obtain the painting trajectory.

8. A graffiti control device in a car, characterized in that: The device comprises: A hand image acquisition module is used to acquire hand images captured by a binocular camera; A hand image analysis module, configured to perform gesture analysis on the hand image to determine a gesture drawing instruction; a drawing picture generating module, configured to generate a drawing trajectory if the gesture drawing instruction is a continuous drawing; render the drawing trajectory into a drawing picture; and control the first display screen to display the drawing picture; The stylized image conversion module is used to start timing from the moment the continuous painting ends and determine whether a picture stylization instruction input by the user is collected within a preset time period; if so, convert the painting picture into a stylized picture according to the picture stylization instruction.

9. A graffiti control device in a vehicle, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the in-vehicle graffiti control method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the in-vehicle graffiti control method according to any one of claims 1 to 7 are implemented.

Images