Paintbrush moving track display method and device, head-mounted equipment and storage medium

Abstract

The embodiment of the invention provides a paintbrush moving track display method and device, head-mounted equipment and a storage medium, and the method comprises the steps: responding to a preset gesture recognized by a camera component of the head-mounted equipment, and displaying a virtual paintbrush associated with the preset gesture; controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a paintbrush movement track of the virtual paintbrush; and displaying the paintbrush moving track. The gesture of the user is recognized for drawing through the camera component of the head-mounted equipment, and the recognition range of the camera component is large, so that the process of drawing through the gesture by the user is not limited to a screen of the equipment, the drawing area of the user is enlarged, and the drawing efficiency is improved. And the head-mounted device can display the virtual paintbrush associated with the preset gesture on the display interface of the head-mounted device and display the paintbrush moving track of the virtual paintbrush, so that the user can conveniently check the painted content in real time, the operability in the painting process is improved, and further, the visibility of the painted content is also improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, head-mounted device and storage medium for displaying a brush movement trajectory. Background Art

[0002] With the development of science and technology, more and more electronic devices support painting functions. However, the current painting functions require users to hold a brush to paint on the screen of the electronic device. The brush is a stylus compatible with the electronic device. For some electronic devices, the user's finger can also be used as a brush. Regardless of whether the user holds the brush to paint on the screen of the electronic device or the user's finger is used as a brush to paint on the screen of the electronic device, it is necessary to touch the screen of the electronic device, and the painting area is limited to the screen of the electronic device. The screens of current electronic devices are relatively small, resulting in a small painting area and low operability during the painting process. Summary of the invention

[0003] The embodiments of the present invention provide a method and apparatus for displaying a brush movement trajectory, a head-mounted device, and a storage medium, which are intended to recognize user gestures for painting through a camera component of a head-mounted device. Since the camera component has a large recognition range, the user is not limited to the device screen during the process of painting through gestures, thereby expanding the user's painting area. In addition, since the head-mounted device will display a virtual brush associated with a preset gesture on its display interface, and display the brush movement trajectory of the virtual brush based on the movement of the preset gesture, it is convenient for the user to view the content painted by the user through the gesture in real time, thereby improving the operability of the painting process.

[0004] In a first aspect, an embodiment of the present invention provides a method for displaying a brush movement trajectory, the method being applied to a head mounted device, the method comprising:

[0005] In response to the camera assembly of the head mounted device recognizing a preset gesture, displaying a virtual brush associated with the preset gesture;

[0006] Controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush;

[0007] Display the brush movement track.

[0008] In a second aspect, an embodiment of the present invention provides a method for displaying a brush movement trajectory, the method being applied to a head mounted device, the method comprising:

[0009] A brush display module, configured to display a virtual brush associated with a preset gesture in response to the camera assembly of the head mounted device recognizing the preset gesture;

[0010] A trajectory determination module, used to control the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush;

[0011] The trajectory display module is used to display the movement trajectory of the brush.

[0012] In a third aspect, an embodiment of the invention further provides a head-mounted device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the method for displaying the brush movement trajectory as described in the first aspect is realized.

[0013] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for displaying the brush movement trajectory as described in the first aspect.

[0014] The embodiment of the present invention provides a method, device, head-mounted device and storage medium for displaying a brush movement trajectory. The embodiment of the present application uses a camera component of a head-mounted device to recognize a user's preset gesture, then displays a virtual brush associated with the preset gesture, controls the movement of the virtual brush through the movement of the preset gesture, thereby obtaining the brush movement trajectory of the virtual brush, and displays the brush movement trajectory, thereby realizing the use of a camera component of a head-mounted device to recognize the user's gesture for painting. Since the camera component has a large recognition range, the user is not limited to the screen of the device during the process of painting through gestures, thereby expanding the user's painting area. Moreover, since the head-mounted device will display the virtual brush associated with the preset gesture on its display interface, and display the brush movement trajectory of the virtual brush based on the movement of the preset gesture, it is convenient for the user to view the content painted by the user through the gesture in real time, thereby improving the operability during the painting process, and further improving the visibility of the painting content. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is a schematic structural block diagram of a head mounted device provided by an embodiment of the present invention;

[0017] Figure 2It is a flowchart of a method for displaying a brush movement trajectory provided by an embodiment of the present invention;

[0018] Figure 3 A reference schematic diagram of a preset gesture provided by an embodiment of the present invention;

[0019] Figure 4 A reference schematic diagram of painting according to a virtual brush provided in an embodiment of the present invention;

[0020] Figure 5 is another flow chart of a method for displaying a brush movement trajectory provided by an embodiment of the present invention;

[0021] Figure 6 It is a reference schematic diagram of the corresponding association information of the brush movement trajectory provided by the embodiment of the present invention;

[0022] Figure 7 It is a reference schematic diagram for mapping a brush movement trajectory in a head mounted device display interface provided by an embodiment of the present invention;

[0023] Figure 8 The present invention is a schematic block diagram of a device for displaying a brush movement trajectory provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] The embodiment of the present invention provides a method, device, head-mounted device and storage medium for displaying a brush movement trajectory. The embodiment of the present invention provides a technical solution.

[0028] In the embodiment of the present invention, the head-mounted device may be AR (Augmented Reality) glasses, an AR helmet, VR glasses, etc.

[0029] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] Figure 1 The present invention is a schematic structural block diagram of a head mounted device provided by an embodiment of the present invention.

[0031] like Figure 1 As shown, the head mounted device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0032] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire head-mounted device 100. The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0033] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0034] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the head-mounted device to which the embodiment of the present invention is applied. The specific head-mounted device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0035] The processor 101 is used to run a computer program stored in the memory 102. The computer program can be set in a head-mounted device. When the head-mounted device executes the computer program, it implements any one of the methods for displaying the brush movement trajectory provided in the embodiments of the present invention.

[0036] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0037] In response to the camera assembly of the head mounted device recognizing a preset gesture, displaying a virtual brush associated with the preset gesture;

[0038] Controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush;

[0039] Display the brush movement track.

[0040] In one embodiment, after the step of displaying the brush movement trajectory, the processor is further configured to implement the following steps when executing the computer program:

[0041] Displaying the associated information corresponding to the brush movement trajectory;

[0042] When a trigger operation on the associated information is detected, first target information corresponding to the associated information is displayed.

[0043] In one embodiment, after the step of displaying the brush movement trajectory, the processor is further configured to implement the following steps when executing the computer program:

[0044] When a voice input command triggering the voice input function is detected, user audio is acquired based on the voice input command, wherein the user audio contains keywords related to the brush movement trajectory;

[0045] The second target information of the brush movement trajectory is obtained based on the user audio, and the second target information is output.

[0046] In one embodiment, the head mounted device includes an optical machine, and before the step of displaying the brush movement trajectory, the processor is further configured to implement the following steps when executing the computer program:

[0047] Obtaining the field of view angle of the optical machine and determining the center point of the field of view angle of the optical machine;

[0048] Determining a display position of the brush movement trajectory in the head mounted device display interface according to the center point of the optical machine field of view;

[0049] The step of displaying the brush movement trajectory comprises:

[0050] The brush movement track is displayed at the display position.

[0051] In one embodiment, before the step of displaying the brush movement trajectory, the processor is further configured to implement the following steps when executing the computer program:

[0052] Determining a display area of ​​the head mounted device display interface and determining a shooting area of ​​the camera assembly;

[0053] Determining a display ratio of the brush movement track in the display interface according to the shooting area and the display area;

[0054] The step of displaying the brush movement trajectory comprises:

[0055] The brush movement trajectory is displayed in the display interface according to the display ratio.

[0056] In one embodiment, after the step of controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain the brush movement trajectory of the virtual paintbrush, the processor is further configured to implement the following steps when executing the computer program:

[0057] Processing the brush movement trajectory based on a preset compensation algorithm to obtain a processed brush movement trajectory;

[0058] The step of displaying the brush movement trajectory comprises:

[0059] The processed brush movement trajectory is displayed.

[0060] In one embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0061] If the duration of the preset gesture stopping the movement is greater than or equal to the preset duration, or if it is detected that the preset gesture is changed to a target gesture corresponding to the generated picture, a picture corresponding to the brush movement trajectory is generated.

[0062] It should be noted that, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the head-mounted device described above can refer to the corresponding process in the following embodiment of the method for displaying the movement trajectory of the brush, and will not be repeated here.

[0063] The following will be combined Figure 1 The head mounted device in the embodiment of the present invention is used to introduce the method for displaying the brush movement trajectory in detail. Figure 1The head mounted device is only used to explain the method for displaying the brush movement trajectory provided in the embodiment of the present invention, but does not constitute a limitation on the application scenario of the method for displaying the brush movement trajectory provided in the embodiment of the present invention.

[0064] See also Figure 2 , Figure 2 It is a flowchart of a method for displaying a brush movement trajectory provided by an embodiment of the present invention.

[0065] like Figure 2 As shown, the method for displaying the brush movement trajectory is applied to a head mounted device, and the method for displaying the brush movement trajectory includes steps S101 to S103.

[0066] Step S101: In response to a camera assembly of the head mounted device recognizing a preset gesture, displaying a virtual brush associated with the preset gesture.

[0067] When the head-mounted device is in a wearing state, if the head-mounted device detects a start-up instruction of the camera component, the head-mounted device starts its camera component based on the start-up instruction. It can be understood that the camera component is a camera in the head-mounted device, or a sensor that can be used for shooting, etc. There are many ways to trigger the start-up instruction, and the embodiments of the present application do not impose specific restrictions on this. For example, the user can trigger the start-up instruction by clicking the camera component icon displayed in the head-mounted device, or by pressing a button in the head-mounted device, or the head-mounted device can trigger the start-up instruction through its built-in red line sensor, IMU sensor and other devices.

[0068] When the head-mounted device starts the camera component, the head-mounted device determines whether a preset gesture is recognized through the camera component. The preset gesture is a pre-set gesture, such as a two-finger pinch gesture, or an "OK" gesture. When the head-mounted device recognizes the preset gesture, the virtual brush associated with the preset gesture is displayed. It can be understood that when the head-mounted device recognizes the preset gesture, it indicates that the head-mounted device is about to start the mid-air painting function. Please refer to Figure 3 , Figure 3 A schematic diagram of a preset gesture in an embodiment of the present invention, through Figure 3 It can be seen that when the user raises his hand and pinches his fingers together, the head-mounted device can recognize the user's two-finger pinch gesture.

[0069] In the embodiment of the present application, the preset gesture for starting the painting function may be one or more. When there are multiple preset gestures, different preset gestures may correspond to different virtual brushes. When there is one preset gesture, the preset gesture may correspond to one or more virtual brushes, and different virtual brushes may correspond to different shapes and / or different virtual brushes may correspond to different thicknesses. Figure 4 , Figure 4 FIG. 1 is a schematic diagram of painting with a virtual brush in an embodiment of the present invention. Figure 4 In the figure, a virtual brush is displayed.

[0070] The process of the head-mounted device recognizing the user gesture through the camera component is as follows: the head-mounted device takes a picture containing the user gesture through the camera component, identifies the key points of the gesture in the picture, which may be the key points of the fingertips and / or the key points of the center of the palm, and performs edge detection on the taken picture containing the user gesture to detect the boundary of the user gesture in the picture, and analyzes the overall shape of the user gesture in the picture or the outline of the user's palm through the identified key points and / or the boundary of the user gesture. In one embodiment, in order to improve the accuracy of recognizing the user gesture, in the process of recognizing the preset gesture, a convolutional neural network or other deep learning model can be used to detect and segment the user's hand area in the picture, so as to separate the user gesture in the picture from the background in the picture.

[0071] After the gesture features of the user's gesture are obtained, the gesture features are input into the pre-trained gesture recognition model. At this time, the gesture output by the gesture recognition model is the user's gesture recognized by the camera component. The gesture output by the gesture recognition model is compared with the preset gesture. If the similarity between the gesture output by the gesture recognition model and the preset gesture is greater than or equal to the preset similarity, it indicates that the head-mounted device recognizes the preset gesture; if the similarity between the gesture output by the gesture recognition model and the preset gesture is less than the preset similarity, it indicates that the head-mounted device does not recognize the preset gesture. The gesture features at least include one or more of the key points of the gesture in the picture, the boundary of the gesture in the picture, the overall shape of the user's gesture, and the outline of the user's palm. The gesture recognition model can be a classification algorithm or a deep learning model. The classification algorithm can be a support vector machine or a decision tree, etc., and the deep learning model can be a convolutional neural network model. This embodiment does not specifically limit the size of the preset similarity. For example, the preset similarity can be set to 70%, 80%, or 85%.

[0072] It is understandable that user gestures change within a certain period of time, that is, user gestures may involve changes in time. At this time, time series analysis can also be performed on the pictures containing user gestures taken by the camera component. For example, a recursive neural network can be used to analyze the pictures from front to back according to the picture acquisition time, so as to accurately identify the user gestures, that is, accurately determine whether the preset gesture is recognized.

[0073] Step S102: Control the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush.

[0074] Step S103: displaying the brush movement trajectory.

[0075] When the head-mounted device recognizes a preset gesture, the head-mounted device can use the camera component to identify the movement of the preset gesture while the user maintains the preset gesture. For easy distinction, the movement trajectory of the preset gesture is defined as the gesture movement trajectory. It can be understood that the gesture movement trajectory is the movement path of the preset gesture when the user maintains the preset gesture. In order to accurately identify the gesture movement trajectory, the head-mounted device can determine a key point of the preset gesture as the recognition reference point of the gesture movement trajectory, such as Figure 3 For the two-finger pinch gesture in the figure, the contact points of the thumb and index finger can be determined as the recognition reference points, and the movement trajectory of the recognition reference points can be determined as the gesture movement trajectory.

[0076] When the head-mounted device recognizes the movement of a preset gesture, it controls the movement of the virtual brush based on the movement of the preset gesture to obtain the brush movement trajectory of the virtual brush. It can be understood that the shapes of the gesture movement trajectory and the brush movement trajectory are the same. When the similarity between the gesture movement trajectory and the brush movement trajectory is greater than or equal to the preset gesture similarity, it can be determined that the shapes of the gesture movement trajectory and the brush movement trajectory are the same; when the similarity between the gesture movement trajectory and the brush movement trajectory is less than the preset gesture similarity, it can be determined that the shapes of the gesture movement trajectory and the brush movement trajectory are different. Among them, the preset gesture similarity can be set to 68%, 75%, or 86%, etc.

[0077] In the process of determining the gesture movement trajectory, the head-mounted device will obtain continuous frames of pictures in the video stream shot by the camera component in real time, record the user's gesture in each frame, and then use the tracking algorithm to track the position of the user's gesture in each frame. According to the order in which each frame is generated, the positions in each frame are connected from front to back, thereby obtaining the gesture movement trajectory, wherein the tracking algorithm can be Kalman filtering, optical flow method or deep learning method, etc. In one embodiment, after obtaining the gesture movement trajectory, the head-mounted device can store the gesture movement trajectory so that the user can view the gesture movement trajectory when needed later.

[0078] When the head mounted device obtains the brush movement track, it will display the brush movement track on its display interface. Figure 4 If the user draws a triangle through a preset gesture, the virtual brush's brush movement trajectory is also a triangle, and the head-mounted device will generate a corresponding triangle image of the brush movement trajectory. In one embodiment, the brush movement trajectory can be displayed in the form of a picture, that is, after the head-mounted device obtains the brush movement trajectory, it displays the brush movement trajectory in the form of a picture on its display interface.

[0079] The embodiment of the present application uses a camera component of a head-mounted device to identify a user's preset gesture, then displays a virtual brush associated with the preset gesture, controls the movement of the virtual brush through the movement of the preset gesture, thereby obtaining a brush movement trajectory of the virtual brush, and displays the brush movement trajectory, thereby realizing painting by identifying the user's gesture through the camera component of the head-mounted device. Since the camera component has a large recognition range, the user is not limited to the device screen during the painting process through gestures, thereby expanding the user's painting area. Moreover, since the head-mounted device will display the virtual brush associated with the preset gesture on its display interface, and display the brush movement trajectory of the virtual brush based on the movement of the preset gesture, it is convenient for the user to view the content painted by the user through gestures in real time, thereby improving the operability during the painting process, and further improving the visibility of the painting content.

[0080] In one embodiment, referring to Figure 5 The method for displaying the brush movement trajectory further includes:

[0081] Step S104, displaying the associated information corresponding to the brush movement trajectory.

[0082] After the head mounted device displays the brush movement trajectory, the head mounted device may display the associated information corresponding to the brush movement trajectory on its display interface, wherein the associated information may be description information of the brush movement trajectory, question information of the brush movement trajectory, or other information related to the brush movement trajectory. Figure 6 , Figure 6 Schematic diagram of the corresponding associated information of the brush movement trajectory in the embodiment of the present invention. Figure 6 In the example, the brush movement trajectory is a triangle, and the associated information is information associated with the triangle, such as “what is the trigonometric formula”, “help me generate a picture”, or “it looks like you drew a triangle”.

[0083] In one embodiment, the display interface for displaying the associated information and the display interface for displaying the brush movement trajectory may be different areas of the same display interface, or may be different display interfaces. When the display interface for displaying the associated information and the display interface for displaying the brush movement trajectory are different display interfaces, the user may control the head mounted device to switch between the display interface for displaying the associated information and the display interface for displaying the brush movement trajectory through an interface switching operation.

[0084] Step S105: when a triggering operation on the associated information is detected, displaying first target information corresponding to the associated information.

[0085] After the head-mounted device displays the brush movement trajectory, the head-mounted device detects the trigger operation corresponding to the associated information. The trigger operation is triggered by the user. When the head-mounted device detects the user's specific voice or specific gesture, it indicates that the head-mounted device detects the trigger operation, and displays the first target information corresponding to the associated information based on the trigger operation. It can be understood that the first target information corresponding to different associated information can be the same or different. In the display interface of the associated information, a voice input button can be displayed, and the user can input the trigger operation through the voice input button, such as inputting the voice "What is the trigonometric formula", at this time the head-mounted device will display the first target information of the associated information "What is the trigonometric formula", and at this time the first target information is the trigonometric formula.

[0086] In one embodiment, the associated information may be in a clickable format. When the head mounted device detects a click operation of the user clicking on the associated information, it indicates that the head mounted device detects a trigger operation of the associated information, and the head mounted device will display the first target information corresponding to the associated information. For example, when the user clicks on the associated information "help me generate a picture", the head mounted device will generate a picture corresponding to the brush movement track.

[0087] In one embodiment, the user may also input other associated information corresponding to the brush movement trajectory by voice or gesture, wherein the other associated information is not currently displayed in the head mounted device display interface. For example, when the brush movement trajectory is a triangle, the other associated information may be "fill the triangle with blue". When the head mounted device detects a triggering operation of the other associated information, the head mounted device will display the first target information of the other associated information, such as displaying a triangle filled with blue.

[0088] In this embodiment, after the head-mounted device displays the brush movement trajectory, the associated information of the brush movement trajectory is displayed, and when a trigger operation on the associated information is detected, the first target information corresponding to the associated information is displayed. Through the first target information, the user can obtain more information corresponding to the brush movement trajectory, rather than just displaying the content drawn by the user through gestures, thereby improving the diversity of painting information display.

[0089] In one embodiment, the method further comprises:

[0090] When a voice input command triggering the voice input function is detected, user audio is acquired based on the voice input command, wherein the user audio contains keywords related to the brush movement trajectory;

[0091] The second target information of the brush movement trajectory is obtained based on the user audio, and the second target information is output.

[0092] In one embodiment, after the head-mounted device displays the brush movement trajectory, if the head-mounted device detects the voice input instruction of the user inputting the voice input function, the head-mounted device can obtain the user audio based on the voice input instruction, wherein the user audio contains keywords related to the brush movement trajectory, and the keywords are used to describe the brush movement trajectory. After the head-mounted device obtains the user audio, the head-mounted device obtains the target information associated with the brush movement trajectory through the user audio. In order to facilitate the distinction from the target information in the above embodiment, in the embodiment of the present application, the target information of the brush movement trajectory obtained by the user audio in this embodiment is recorded as the second target information, and the target information in the above embodiment is recorded as the first target information. When the head-mounted device obtains the second target information, the head-mounted device will output the second target information. It can be understood that the output mode of the second target information may be voice, text and / or picture, and the output mode and specific content of the second target information depend on the user audio. In this embodiment, the content of the second target information is not limited.

[0093] For example, refer to Figure 6 , when the user clicks Figure 6 When the user clicks to speak in the relevant position, the voice input function can be triggered. At this time, if the user speaks, the head-mounted device can collect the user audio. For example, when the brush moves along a tree, the user audio can be "I want a green tree", the keyword is "tree", and the second target information is a green tree, so the output method of the second target information is a picture.

[0094] In this embodiment, after generating a brush movement trajectory, the user can obtain the second target information of the brush movement trajectory through voice input instructions and output the second target information. Through the second target information, the user can obtain more information corresponding to the brush movement trajectory, rather than just displaying the content drawn by the user through gestures, thereby improving the diversity of painting information display.

[0095] In one embodiment, before step S103, the head mounted device includes an optical machine, and the method further includes:

[0096] Obtaining the field of view angle of the optical machine and determining the center point of the field of view angle of the optical machine;

[0097] Determining a display position of the brush movement trajectory in the head mounted device display interface according to the center point of the optical machine field of view;

[0098] The step S103 includes:

[0099] The brush movement track is displayed at the display position.

[0100] In one embodiment, the head-mounted device is a device that includes an optical machine. When the head-mounted device recognizes a preset gesture through a camera assembly, the head-mounted device obtains the field of view of its optical machine and determines the center point of the field of view of the optical machine. After the head-mounted device determines the center point of the field of view of the optical machine, the head-mounted device determines the display position of the brush movement trajectory in its display interface according to the center point of the field of view of the optical machine, and displays the brush movement trajectory in the determined display position. It can be understood that since the field of view of the camera assembly of the head-mounted device is different from the field of view of the optical machine, and the camera assembly and the optical machine are located at different positions, the display position of the brush movement trajectory under the field of view of the camera assembly and the display position under the field of view of the optical machine are different. Please refer to Figure 7 ,exist Figure 7 In the figure, the small rectangle is the field of view (FOV) of the optical machine, and the large rectangle is the field of view of the camera component (i.e., the camera FOV). It can be understood that the field of view of the camera component is the recognition range of the camera component. The field of view of the optical machine is the display range of the head-mounted device display interface.

[0101] In one embodiment, the center point of the brush movement trajectory can be determined as the display position of the brush movement trajectory, and in other embodiments, other positions of the brush movement trajectory can also be determined as the display position of the brush movement trajectory. Since the display position of the brush movement trajectory under the field of view of the camera component and the display position under the optical machine field of view are different, in order to make the brush movement trajectory seen by the user wearing the head-mounted device displayed in the middle of the display interface, this embodiment determines the center point of the optical machine field of view as the display position of the brush movement trajectory, so that the center point of the brush movement trajectory coincides with the center point of the optical machine field of view, thereby controlling the brush movement trajectory to be displayed in the middle of the head-mounted device display interface.

[0102] This embodiment determines the center point of the optical machine field of view of the head-mounted device, and determines the center point of the optical machine field of view as the display position of the brush movement trajectory. After obtaining the brush movement trajectory, the brush movement trajectory is displayed according to the center point of the optical machine field of view, so that the center point of the brush movement trajectory and the center point of the optical machine field of view coincide with each other, and the brush movement trajectory is displayed in the middle of the head-mounted device display interface, which is convenient for users wearing the head-mounted device to view the brush movement trajectory, thereby improving the intelligence of the brush movement trajectory display.

[0103] In one embodiment, before step S103, the head mounted device includes a camera assembly, and the method further includes: determining a display area of ​​a display interface of the head mounted device, and determining a shooting area of ​​the camera assembly;

[0104] Determining a display ratio of the brush movement track in the display interface according to the shooting area and the display area;

[0105] The step S103 includes:

[0106] The brush movement trajectory is displayed in the display interface according to the display ratio.

[0107] When the head-mounted device recognizes a preset gesture through the camera component, the head-mounted device obtains the field of view angle of the camera component and the field of view angle of its optical machine. For the sake of distinction, this embodiment records the field of view angle of the optical machine as the first field of view angle and the field of view angle of the camera component as the second field of view angle. The head-mounted device determines the display area of ​​its display interface based on the first field of view angle, and determines the photographing area of ​​the camera component based on the second field of view angle. In the head-mounted device, the display area corresponding to the first field of view angle and the display area corresponding to the second field of view angle can be pre-stored. The head-mounted device can determine the display area of ​​the display interface through the first field of view angle, and can determine the photographing area of ​​the camera component through the second field of view angle.

[0108] In one embodiment, the display area and the shooting area of ​​the head mounted device display interface may also be directly stored in the head mounted device, and the head mounted device may directly obtain the display area and the shooting area. The display area of ​​the display interface carries an optical machine identifier, and the shooting area carries a camera identifier, and the display area and the shooting area of ​​the display interface may be distinguished by the optical machine identifier and the camera identifier.

[0109] After the head mounted device obtains the shooting area and the display area, the head mounted device divides the display area by the shooting area to obtain the display ratio of the brush movement track in the display interface, scales the brush movement track according to the display ratio to obtain the scaled brush movement track, and displays the scaled brush movement track in the display interface. It can be understood that for most head mounted devices, the second field of view angle is greater than the first field of view angle, that is, the shooting area is greater than the display area, and therefore, the scaled brush movement track is smaller than the brush movement track before scaling.

[0110] It is understandable that the embodiments of the present application are not limited to the method of calculating the display ratio, such as also based on the optical machine center point ( Figure 7 O1 in the image) and the center point of the camera assembly ( Figure 7 The area of ​​the rectangle based on the center point coordinates can be calculated by referring to the relevant technology, which will not be described in detail here.

[0111] In this embodiment, the head-mounted device first determines the shooting area of ​​the shooting component and the display area of ​​the display interface, determines the display ratio based on the shooting area and the display area, and then displays the brush movement track in the display interface according to the display ratio. Since the shooting area is larger than the display area, if the brush movement track identified by the camera component is directly displayed in the display interface, there may be a situation where the brush movement track cannot be fully displayed in the display interface. However, through the display ratio, the brush movement track will be scaled according to the display ratio to make the brush movement track adapt to the size of the head-mounted display interface, so that the brush movement track is fully displayed in the display interface, which improves the intelligence of the brush movement track display.

[0112] In one embodiment, when the head-mounted device scales the brush movement trajectory according to the zoom ratio to obtain the scaled brush movement trajectory, the head-mounted device determines the center point of its optical machine field of view as the display position of the scaled brush movement trajectory, and displays the scaled brush movement trajectory based on the determined display position, so that the scaled brush movement trajectory is displayed in the middle position of the head-mounted device display interface, further improving the intelligence of the brush movement trajectory display.

[0113] In one embodiment, after step S102, the method further includes: processing the brush movement trajectory based on a preset compensation algorithm to obtain a processed brush movement trajectory.

[0114] Step S103 includes: displaying the processed brush movement trajectory.

[0115] In one embodiment, after the head-mounted device obtains the brush movement trajectory, the head-mounted device obtains a preset compensation algorithm, processes the brush movement trajectory based on the preset compensation algorithm, thereby obtaining a processed brush movement trajectory, and displays the processed brush movement trajectory. The preset compensation algorithm can be pre-stored in the head-mounted device, or can be obtained by the head-mounted device from a cloud server or other electronic device when necessary. It can be understood that the trajectory of the processed brush movement trajectory will be more coherent and look more beautiful compared to the brush movement trajectory before processing. Exemplarily, if the triangle of the brush movement trajectory before processing has two lines that are not connected together, that is, the triangle is not a closed triangle, then the processed brush movement trajectory is a closed triangle. If the triangle of the brush movement trajectory before processing has a side length that is not a straight line, then all the sides of the triangle of the brush movement trajectory after processing are straight lines.

[0116] The compensation algorithm is described below:

[0117] In one embodiment, a parameter estimation model is first established, and establishing the parameter estimation model includes the following steps:

[0118] Step 1: Establish a motion model, wherein the motion model can be based on the equations of kinematics and dynamics in physics. Common motion models include but are not limited to kinematic models and dynamic models. The kinematic model describes the motion of an object without considering its mass and force. For example, the motion is described by displacement, velocity, and acceleration. The dynamic model considers the influence of force and mass, for example, using Newton's second law to describe the relationship between the acceleration and force of an object.

[0119] For the movement of the fingers in the preset gesture, a simple kinematic model or a more complex dynamic model can be used. The simple kinematic model is used to describe the linear or curved movement of the finger, such as using the position-time relationship to predict the position of the finger. The dynamic model can take into account factors such as the mass of the finger, friction, and applied force.

[0120] Step 2: Establish a finger motion model based on the motion model in step 1, wherein the finger motion model includes at least one or more of a kinematic model, a dynamic model, and a damping and elasticity model. The motion model can be used to characterize a linear motion model and a circular motion model, wherein the linear motion model is used to describe the linear motion of a finger in a certain direction. If the finger moves along a curved trajectory, a circular motion model can be used. The dynamic model can be Newton's equation of motion, through which the relationship between force and mass can be used to describe the finger motion. The damping and elasticity model can take into account the effects of friction and elastic force on finger motion.

[0121] Step 3: Parameter estimation based on the established finger motion model. After the finger motion model is established, the finger movement data can be actually measured to estimate motion parameters such as velocity, acceleration, friction coefficient, etc. Then, based on the motion parameters, the least squares method, Kalman filtering and other algorithms are used to estimate the model parameters of the finger motion model, so that the error between the predicted value output by the finger motion model and the actual data is minimized, thereby training a finger motion model that can be applied.

[0122] Step 4: Based on the finger motion model obtained in step 3, trajectory prediction and motion compensation of the brush movement trajectory are performed, wherein trajectory prediction is to use the finger motion model to predict the future trajectory. For example, if the current movement speed and acceleration of the finger are known, the future position of the finger can be predicted. Motion compensation is to adjust the finger movement to correct the deviation between the trajectory prediction and the actual trajectory. For example, the acceleration of the finger or the force applied is adjusted in real time according to the actual data to maintain the consistency of the brush movement trajectory.

[0123] It is understandable that in other embodiments, the preset compensation algorithm may also be a model trained by neural network, deep learning, etc. In this embodiment, there is no specific restriction on the type of compensation algorithm, as long as the brush movement trajectory after processing can be made more coherent than the brush movement trajectory before processing.

[0124] After obtaining the brush movement trajectory, this embodiment processes the brush movement trajectory through a preset compensation algorithm to obtain a processed brush movement trajectory, and displays the processed brush movement trajectory. Since the processed brush movement trajectory is more coherent than the brush movement trajectory before processing, the coherence of the displayed brush movement trajectory is improved, that is, the aesthetics of the user's painting is improved.

[0125] In one embodiment, the method further comprises:

[0126] If the duration of the preset gesture stopping the movement is greater than or equal to the preset duration, or if it is detected that the preset gesture is changed to a target gesture corresponding to the generated picture, a picture corresponding to the brush movement trajectory is generated.

[0127] In this embodiment, when the head-mounted device obtains the brush movement trajectory and displays the brush movement trajectory in its display interface, the brush movement trajectory is not displayed in the display interface in the form of a picture. After the head-mounted device displays the brush movement trajectory, the head-mounted device obtains the duration of the preset gesture stopping movement, and determines whether the duration of the preset gesture stopping movement is greater than or equal to the preset duration, wherein the size of the preset duration can be set as needed, such as 2 seconds, 3 seconds or 5 seconds. If the duration of the preset gesture stopping movement is greater than or equal to the preset duration, it indicates that the user has completed the painting, and the head-mounted device can generate a picture corresponding to the brush movement trajectory; if the duration of the preset gesture stopping movement is less than the preset duration, the head-mounted device continues to determine whether the duration of the preset gesture stopping movement is greater than or equal to the preset duration, or determines whether the target gesture is detected.

[0128] After the head-mounted device displays the brush movement trajectory, the head-mounted device can also determine whether to detect that the user's preset gesture has changed to the target gesture corresponding to the generated image. It can be understood that the preset gesture and the target gesture are different. When the head-mounted device detects the target gesture, the head-mounted device generates a picture of the brush movement trajectory based on the target gesture. If the head-mounted device does not detect the target gesture, the head-mounted device continues to determine whether the duration of the preset gesture stopping the movement is greater than or equal to the preset duration, or determines whether the target gesture is detected.

[0129] After the head-mounted device generates a picture corresponding to the brush movement trajectory, the head-mounted device can store the picture and / or display the picture in its display interface.

[0130] This embodiment improves the intelligence of generating pictures corresponding to the brush movement trajectory by automatically generating pictures corresponding to the brush movement trajectory when the duration of stopping motion of the preset gesture is greater than or equal to the preset duration, or when it is detected that the preset gesture is changed to the target gesture corresponding to the generated picture.

[0131] See also Figure 8 , Figure 8 The present invention is a schematic block diagram of a device for displaying a brush movement trajectory provided by an embodiment of the present invention.

[0132] like Figure 8 As shown, the display device 800 for the brush movement trajectory is applied to a head mounted device, and the display device 800 for the brush movement trajectory includes:

[0133] A brush display module 810, configured to display a virtual brush associated with a preset gesture in response to the camera assembly of the head mounted device recognizing the preset gesture;

[0134] A trajectory determination module 820, configured to control the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush;

[0135] The trajectory display module 830 is used to display the brush movement trajectory.

[0136] In one embodiment, the device 800 for displaying the brush movement trajectory further includes:

[0137] A related information display module, used to display related information corresponding to the brush movement trajectory;

[0138] The target information display module is used to display the first target information corresponding to the associated information when a trigger operation on the associated information is detected.

[0139] In one embodiment, the device 800 for displaying the brush movement trajectory further includes:

[0140] An audio acquisition module, configured to acquire user audio based on a voice input instruction that triggers a voice input function after a voice input instruction is detected, wherein the user audio contains keywords related to the brush movement trajectory;

[0141] A target information output module is used to obtain second target information of the brush movement trajectory based on the user audio and output the second target information.

[0142] In one embodiment, the head mounted device includes an optical machine, and the display device 800 for the brush movement trajectory further includes:

[0143] A center point determination module, used to obtain the field of view angle of the optical machine and determine the center point of the field of view angle of the optical machine;

[0144] A display position determination module, used to determine the display position of the brush movement trajectory in the display interface of the head mounted device according to the center point of the optical machine field of view;

[0145] The trajectory display module 830 is further configured to display the brush movement trajectory at the display position.

[0146] In one embodiment, the device 800 for displaying the brush movement trajectory further includes:

[0147] An area determination module, used to determine the display area of ​​the head mounted device display interface and determine the shooting area of ​​the camera assembly;

[0148] A display ratio determination module, used to determine the display ratio of the brush movement track in the display interface according to the shooting area and the display area;

[0149] The trajectory display module 830 is further configured to display the brush movement trajectory in the display interface according to the display ratio.

[0150] In one embodiment, the device 800 for displaying the brush movement trajectory further includes:

[0151] A trajectory processing module, used for processing the brush movement trajectory based on a preset compensation algorithm to obtain a processed brush movement trajectory;

[0152] The trajectory display module 830 is also used to display the processed brush movement trajectory.

[0153] In one embodiment, the device 800 for displaying the brush movement trajectory further includes:

[0154] The image generation module is used to generate an image corresponding to the brush movement trajectory if the duration of the preset gesture stopping movement is greater than or equal to the preset duration, or if it is detected that the preset gesture is changed to a target gesture corresponding to the generated image.

[0155] It should be noted that, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the display device of the brush movement trajectory described above can refer to the corresponding process in the aforementioned embodiment of the display method of the brush movement trajectory, and will not be repeated here.

[0156] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any method for displaying a brush movement trajectory provided in the description of the embodiment of the present invention.

[0157] The storage medium may be an internal storage unit of the head-mounted device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted device. The storage medium may also be an external storage device of the head-mounted device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the head-mounted device.

[0158] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0159] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0160] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for displaying a brush movement trajectory, characterized in that: The method is applied to a head mounted device, and the method comprises: In response to the camera assembly of the head mounted device recognizing a preset gesture, displaying a virtual brush associated with the preset gesture; Controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush; Display the brush movement track.

2. The method for displaying the brush movement trajectory according to claim 1, characterized in that: After the step of displaying the brush movement trajectory, the method further includes: Displaying the associated information corresponding to the brush movement trajectory; When a trigger operation on the associated information is detected, first target information corresponding to the associated information is displayed.

3. The method for displaying the brush movement trajectory according to claim 1, characterized in that: After the step of displaying the brush movement trajectory, the method further includes: When a voice input command triggering the voice input function is detected, user audio is acquired based on the voice input command, wherein the user audio contains keywords related to the brush movement trajectory; The second target information of the brush movement trajectory is obtained based on the user audio, and the second target information is output.

4. The method for displaying the brush movement trajectory according to claim 1, characterized in that: The head mounted device includes an optical machine, and before the step of displaying the movement trajectory of the brush, further includes: Obtaining the field of view angle of the optical machine and determining the center point of the field of view angle of the optical machine; Determining a display position of the brush movement trajectory in the head mounted device display interface according to the center point of the optical machine field of view; The step of displaying the brush movement trajectory comprises: The brush movement track is displayed at the display position.

5. The method for displaying the brush movement trajectory according to claim 1, characterized in that: Before the step of displaying the brush movement trajectory, the method further includes: Determining a display area of ​​the head mounted device display interface and determining a shooting area of ​​the camera assembly; Determining a display ratio of the brush movement track in the display interface according to the shooting area and the display area; The step of displaying the brush movement trajectory comprises: The brush movement trajectory is displayed in the display interface according to the display ratio.

6. The method for displaying the brush movement trajectory according to claim 1, characterized in that: After the step of controlling the movement of the virtual paintbrush based on the movement of the preset gesture to obtain the brush movement trajectory of the virtual paintbrush, the method further includes: Processing the brush movement trajectory based on a preset compensation algorithm to obtain a processed brush movement trajectory; The step of displaying the brush movement trajectory comprises: The processed brush movement trajectory is displayed.

7. The method for displaying the brush movement trajectory according to claim 1, characterized in that: The method further comprises: If the duration of the preset gesture stopping the movement is greater than or equal to the preset duration, or if it is detected that the preset gesture is changed to a target gesture corresponding to the generated picture, a picture corresponding to the brush movement trajectory is generated.

8. A device for displaying a brush movement trajectory, characterized in that: The device for displaying the movement trajectory of the brush is applied to a head mounted device, and the device for displaying the movement trajectory of the brush comprises: A brush display module, configured to display a virtual brush associated with a preset gesture in response to the camera assembly of the head mounted device recognizing the preset gesture; A trajectory determination module, used to control the movement of the virtual paintbrush based on the movement of the preset gesture to obtain a brush movement trajectory of the virtual paintbrush; The trajectory display module is used to display the movement trajectory of the brush.

9. A head mounted device, characterized in that: The head-mounted device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the method for displaying the brush movement trajectory as described in any one of claims 1 to 7 is realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for displaying the brush movement trajectory according to any one of claims 1 to 7.

Images