Method and device for detecting lip state

[0016] When implementing the present invention, the applicant found that when performing lip state detection, the detection can be performed through changes in the received voice data or additional external sensors such as infrared sensors. However, these methods require additional equipment or sensors and cannot pass The pure visual information of the image is used to detect the lip status.

[0017] The embodiment of the present invention provides a method and device for detecting lip status, which can effectively detect the status of the lip region, has high recognition accuracy and low cost, and can effectively remove the influence of noise data such as silent frames on the lip recognition result and reduce Interference, improve data processing efficiency.

[0018] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0019] The following will be attached figure 1 Attached figure 2 The method for detecting the state of the lips shown in the exemplary embodiment of the present invention is introduced.

[0020] See figure 1 , Is a flowchart of a method for detecting lip status according to an embodiment of the present invention. Such as figure 1 As shown, it can include:

[0021] S101: Perform lip region detection on a target image, and acquire a lip region image included in the target image.

[0022] In specific implementation, face detection may be performed on the target image first, if a face area is detected, then lip area detection is performed on the detected face area, and the lip area image included in the target image is obtained. This application does not limit the method of obtaining the lip region image, and the method provided in the prior art can be used.

[0023] In some embodiments, the performing lip region detection on the target image, and acquiring the lip region image included in the target image includes: acquiring the Nth frame image as the target image; acquiring the lip region detection result of the NMth frame image , Take the area range corresponding to the lip area detection result of the NM-th frame image as the effective range; perform lip area detection in the area corresponding to the target image and the effective range; where N is an integer greater than 1, M Is a positive integer less than N. For example, in the first frame of image, the lip region is detected for the whole face, and the lip region detection result is obtained, and the detection result includes the range of the lip region. The second frame of image may not detect the whole face, but on the basis of the detection result of the previous frame, the range corresponding to the detection result is obtained as the effective range, and the lip region is detected within the effective range, which effectively improves Processing efficiency, and improve the real-time processing.

[0024] S102: Determine multiple key feature points of the lip region image.

[0025] It should be noted that the multiple key feature points are points that can highly characterize the image features of the lip region. For example, when acquiring multiple key feature points, first perform edge detection on the lip region image to determine the edge contour of the lip region. The edge contour includes an outer lip edge line and an inner lip edge line. Multiple feature points can be selected as key feature points on the outer lip edge line and the inner lip edge line. For example, the multiple key feature points may include the highest point and the lowest point in the center of the outer lip contour line, the two points corresponding to the left and right outer corners of the outer lip contour line, and the two corresponding left and right inner corners of the inner lip contour line. Points, at least two key feature points of the upper and lower feature points corresponding to the middle position on the inner lip contour line. Of course, other feature points can also be selected.

[0026] S103: Determine the state of the lip region according to the feature values ​​corresponding to the multiple key feature points; the state of the lip region includes an open state or a closed state.

[0027] In specific implementation, the feature value corresponding to the key feature point includes at least one of a height value, a slope value, or an area value. The determining the state of the lip region according to the feature values ​​corresponding to the plurality of key feature points includes: determining the lip region according to at least one of a height value, a slope value, or an area value of the plurality of key feature points The state of the region.

[0028] In some possible implementation manners, the determining the state of the lip region according to the feature values ​​of the plurality of key feature points includes: determining the height difference of at least two key feature points in the plurality of key feature points ; If the height difference is greater than or equal to the set height threshold, it is determined that the state of the lip region is an open state. For example, the upper and lower two points at the middle position of the contour line of the outer lip of the lip region can be selected as key feature points, and the height difference between the two key feature points can be obtained. The height difference is the difference between the height values ​​of the two feature points. The height value of the two feature points may specifically be the value of the ordinate of the feature point. When a person is speaking, the height difference between the upper and lower points of the middle of the lips will change. Generally, the height difference will increase from closed to open. Therefore, the opening and closing state of the lips can be determined according to the changes in the height difference of the feature points in the middle of the lips. Of course, when it is determined that the height difference between the upper and lower points in the middle of the inner lip is greater than or equal to the first threshold and the height difference between the upper and lower points in the middle of the outer lip is greater than or equal to the second threshold, it can be determined that the lip area is in an open state. . The first threshold and the second threshold can be set based on experience or needs.

[0029] Since the thickness of each person's lips is different, in order to be able to adapt to different groups of people, the state of the lip region can be determined by detecting the changes in the feature values ​​of key feature points in the lip region. For example, in some possible implementation manners, the determining the state of the lip region according to the feature values ​​of the plurality of key feature points includes: determining that at least one of the current Nth frame images of the plurality of key feature points is The height difference between two key feature points is taken as the first height difference; the height difference of the at least two key feature points in the NM-th frame image is determined as the second height difference; if the first height difference is compared with the second height difference The change value of is greater than or equal to the set height change threshold, and it is determined that the state of the lip region is an open state. In this implementation manner, the state of the lip region can be determined by the change in height difference between the feature points of different frame images. For example, if the current frame is the Nth frame of the lip region image, the height difference between the first key feature point and the second key feature point can be obtained as the first difference; in the NMth frame image, the first key feature The height difference between the point and the second key feature point is used as the second difference; if the change value of the first difference and the second difference is greater than or equal to the set height change threshold, the state of the lip area is determined to be open status. Among them, M is a positive integer less than N. For example, if the current frame is the Nth frame, the previous frame or the previous two frames can be obtained for comparison.

[0030] In some possible implementations, the state of the lip region can also be determined by the slope between two feature points. The determining the state of the lip region according to the feature values ​​of the plurality of key feature points includes: determining the slope value of a straight line formed by at least two key feature points in the plurality of key feature points; if the slope is If the value is greater than or equal to the set slope threshold, it is determined that the state of the lip region is an open state. For example, when a person speaks from a closed mouth to an open mouth, the slope of the line connecting the corners of the mouth and the center of the lips becomes larger, so the opening and closing state can be determined based on the slope value. For example, the slopes of 4 straight lines can be determined according to the feature points corresponding to the left and right corners of the mouth and the upper and lower points in the middle of the lips. If the slope value of any one or more straight lines meets the condition, it can be determined that the state of the lip region is an open state. Due to the difference in the shape of the lips of each person, the state of the lip area can be determined according to the slope of the line of the two characteristic points of the lips in the lip area. Of course, the state of the lip region can also be determined by combining the change in the slope values ​​of the feature points of the inner lips and the feature points of the outer lips.

[0031] In some possible implementations, the state of the lip region can also be determined according to the change in the area of ​​the figure formed by the feature points. The determining the state of the lip region according to the feature values ​​of the plurality of key feature points includes: determining the area value of a figure formed by at least three feature points in the plurality of key feature points; if the area value is If the area is greater than or equal to the set area threshold, it is determined that the state of the lip region is an open state. For example, when a person speaks, the area of ​​the figure formed between the corners of the inner lips and the center of the inner lips will change from zero to non-zero. Therefore, if the area value of the graphic formed by at least three feature points is greater than or equal to the set area threshold, the state of the lip region is determined to be the open state. Of course, at least four feature points can also be selected. If the area of ​​the figure formed by the at least four feature points is greater than or equal to the set threshold, the state of the lip region is determined to be the open state. Of course, the area change of the inscribed triangle formed by at least 3 characteristic points of the outer lip or the inscribed quadrilateral formed by at least four characteristic points can also be used to determine whether the lip region is in an open state.

[0032] It should be noted that the above-mentioned manner of determining the state of the lip region by the feature values ​​or changes of the feature values ​​of multiple key feature points of the lip region can be performed on a single frame of images or on multiple frames of images.

[0033] In other embodiments, the state of the lip region can also be determined in combination with the changes between image frames. For example, when a person is speaking, different image frames will change. If the user is in a silent state, the consecutive multiple image frames collected may not change. Therefore, the state of the lip region can be determined by the difference between the image frames to eliminate noise such as silent frames.

[0034] For example, the Nth frame image can be used as the current lip region image to obtain the N-Mth frame lip region image; the frame difference between the current lip region image and the N-Mth frame lip region image is determined. Among them, M is a positive integer less than N. Of course, the state of the lip region can also be determined by the change of the inter-frame difference of consecutive multiple frames of images. For example, the inter-frame difference can be calculated on the lip region images in the continuous 3-5 frames of video stream, and whether the user may be in the open state can be determined by whether the pixel value change is greater than or equal to a certain threshold. The determining the state of the lip region according to the feature values ​​corresponding to the plurality of key feature points includes: determining the state of the lip region according to the feature values ​​corresponding to the plurality of key feature points and the inter-frame difference . That is, when the feature value corresponding to the key feature point satisfies the set condition and the frame difference of the images of different lip regions is greater than or equal to the set threshold, it is determined that the lip region is in the open state. The judgment of whether the characteristic value corresponding to the key characteristic point meets the set condition can be realized by referring to the aforementioned method.

[0035] In some embodiments, the method further includes: extracting a histogram of orientation gradients of the lip region (Histogram of Oriented Gridients in English, HOG for short) features, and determining the lip region according to the HOG features State classification result; the determining the state of the lip region according to the feature values ​​corresponding to the multiple key feature points includes: according to the feature values ​​corresponding to the multiple key feature points, the classification result, and the lip region At least two of the frame differences of the image determine the state of the lip region. It should be noted that the classification result of the state of the lip region can be output according to the HOG feature. Therefore, the state of the lip region can be determined according to at least two of the feature value of the key feature point, the classification result, and the difference between frames of different images.

[0036] In order to facilitate those skilled in the art to more clearly understand the implementation of the present invention in a specific scenario, the following describes the implementation of the present invention with a specific example. It should be noted that this specific example is only to enable those skilled in the art to understand the present invention more clearly, but the embodiments of the present invention are not limited to this specific example.

[0037] See figure 2 , Is a flowchart of a method for detecting lip status according to an embodiment of the present invention. Such as figure 2 As shown, it can include:

[0038] S201: Perform lip region detection on a target image, and acquire a lip region image included in the target image.

[0039] S202: Extract the HOG feature of the lip region image, and determine the classification result of the state of the lip region according to the HOG feature.

[0040] It should be noted that the HOG feature of the sample data can be extracted, and a support vector machine (English full name Support Vector Machine, English abbreviated SVM) classifier can be trained using the open/closed state label and the HOG feature. After the lip region image of the target image is acquired, the HOG feature of the lip region image is extracted and input into the trained SVM classifier to obtain the classification result of the opening and closing state. If the classification result is an open state, continue to execute S203.

[0041] S203: Determine multiple key feature points of the lip region image, and determine the state of the lip region according to feature values ​​corresponding to the multiple key feature points.

[0042] For specific implementation, you can refer to figure 1 The aforementioned S103 is realized. For example, you can input a video stream to detect the height difference of the upper and lower 4 points of the lip area and the change of the inscribed quadrilateral area of ​​the 8 feature points of the lip to determine whether the lip area is in an open state.

[0043] S204. Acquire a previous frame of the lip region image of the current lip region image, and determine the frame difference between the current lip region image and the previous frame of the lip region image; if the frame difference is greater than or equal to a set threshold, determine The lip region is in an open state.

[0044] It should be noted that S202 to S204 are not necessarily executed sequentially, and can be executed in reverse or in parallel, which is not limited here. In addition, in specific implementation, after S201 is executed, only S202 and S203 may be executed, or only S203 and S204 may be executed. Those skilled in the art may determine the state of the lip region according to at least two of the feature values ​​corresponding to multiple key feature points of the lip region, the HOG feature classification result, and the frame difference of the lip region image as needed.

[0045] It should be noted that, in order to improve the accuracy of lip state recognition, you can also use the Long Short-Term Memory network (English full name Long Short-Term Memory, English abbreviation LSTM) model or hidden Markov model (HMM) to the lip region Image sequence is classified, but the algorithm complexity of this method is relatively high, and it cannot meet the real-time requirements well.

[0046] The following describes the equipment and devices corresponding to the lip language recognition method provided by the embodiment of the present invention. The settings of each unit or module of the device can refer to Figure 1 to Figure 2 The method shown is implemented.

[0047] See image 3 , Is a schematic diagram of a lip state detection device provided by an embodiment of the present invention.

[0048] A lip state detection device 300 includes:

[0049] The detection unit 301 is configured to perform lip region detection on a target image, and obtain a lip region image included in the target image. Among them, the specific implementation of the detection unit 301 can refer to figure 1 Step 101 of the illustrated embodiment is implemented.

[0050] The feature point determining unit 302 is configured to determine multiple key feature points of the lip region image. Among them, the specific implementation of the feature point determining unit 302 can refer to figure 1 Step 102 of the illustrated embodiment is implemented.

[0051] The state determining unit 303 is configured to determine the state of the lip region according to the feature values ​​corresponding to the multiple key feature points; the state of the lip region includes an open state or a closed state. Among them, the specific implementation of the state determining unit 303 can refer to figure 1 Step 103 of the illustrated embodiment is implemented.

[0052] In some embodiments, the state determining unit is specifically configured to determine the state of the lip region according to at least one of a height value, a slope value, or an area value corresponding to the plurality of key feature points.

[0053] In some embodiments, the state determining unit includes:

[0054] The first determining unit is configured to determine the height difference of at least two key feature points among the plurality of key feature points; if the height difference is greater than or equal to the set height threshold, determining that the state of the lip region is Open state.

[0055] In some embodiments, the state determining unit includes:

[0056] The second determining unit is configured to determine the height difference of at least two key feature points in the current N-th frame image among the plurality of key feature points as the first height difference; determine that the at least two key feature points are in the NM image The height difference in is regarded as the second height difference; if the change value of the first height difference and the second height difference is greater than or equal to the set height change threshold, it is determined that the state of the lip region is an open state. Among them, M is a positive integer less than N.

[0057] In some embodiments, the state determining unit includes:

[0058] The third determining unit is used to determine the slope value of the straight line formed by at least two key feature points among the plurality of key feature points; if the slope value is greater than or equal to the set slope threshold value, determine the lip region The state is open.

[0059] In some embodiments, the state determining unit includes:

[0060] The fourth determining unit is used to determine the area value of the figure formed by at least three feature points among the plurality of key feature points; if the area value is greater than or equal to the set area threshold, determine the area value of the lip region The state is an open state.

[0061] In some embodiments, the device further includes:

[0062] The inter-frame difference calculation unit is used to take the Nth frame image as the current lip region image to obtain the current NMth frame lip region image; where M is a positive integer less than N; determine the current lip region image and the NMth frame The difference between frames of the lip region image;

[0063] The state determining unit includes:

[0064] The fifth determining unit is configured to determine the state of the lip region according to the feature values ​​corresponding to the multiple key feature points and the difference between frames.

[0065] In some embodiments, the device further includes:

[0066] The HOG feature classification unit is configured to extract HOG features of the directional gradient histogram of the lip region, and determine the classification result of the state of the lip region according to the HOG feature;

[0067] The state determining unit includes:

[0068] The sixth determining unit is configured to determine the state of the lip region according to at least two of the feature values ​​corresponding to the multiple key feature points, the classification result, and the inter-frame differences of different lip region images.

[0069] In some embodiments, the detection unit is specifically configured to:

[0070] Obtain the Nth frame image as the target image; acquire the lip region detection result of the NM frame image, and use the region range corresponding to the lip region detection result of the NM frame image as the effective range; in the target image and the effective range Lip region detection is performed in the corresponding region; where N is an integer greater than 1, and M is a positive integer less than N.

[0071] Among them, the setting of each unit or module of the device of the present invention can refer to Figure 1 to Figure 2 The method shown is implemented, and will not be repeated here.

[0072] See Figure 4 , Is a block diagram showing a device for detecting lip status according to an exemplary embodiment. See Figure 4 , Is a block diagram showing a device for detecting lip status according to an exemplary embodiment. For example, the apparatus 400 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0073] Reference Figure 4 , The device 400 may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input/output (I/O) interface 412, a sensor component 414, and a communication component 416.

[0074] The processing component 402 generally controls the overall operations of the device 400, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0075] The memory 404 is configured to store various types of data to support the operation of the device 400. Examples of these data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

[0076] The power supply component 406 provides power to various components of the device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 400.

[0077] The multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of the touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and/or a rear camera. When the device 400 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0078] The audio component 410 is configured to output and/or input audio signals. For example, the audio component 410 includes a microphone (MIC). When the device 400 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 further includes a speaker for outputting audio signals.

[0079] The I/O interface 412 provides an interface between the processing component 402 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0080] The sensor component 414 includes one or more sensors for providing the device 400 with various aspects of status assessment. For example, the sensor component 414 can detect the on/off status of the device 400 and the relative positioning of components. For example, the component is the display and keypad of the device 400. The sensor component 414 can also detect the position change of the device 400 or a component of the device 400. , The presence or absence of contact between the user and the device 400, the orientation or acceleration/deceleration of the device 400, and the temperature change of the device 400. The sensor component 414 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0081] The communication component 416 is configured to facilitate wired or wireless communication between the apparatus 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 414 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 414 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0082] In an exemplary embodiment, the apparatus 400 may be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing equipment (DSPD), programmable logic devices (PLD), field programmable A gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.

[0083] Specifically, an embodiment of the present invention provides a lip state detection device 400, which includes a memory 404 and one or more programs. One or more programs are stored in the memory 404 and configured to be configured by one or more programs. The execution of the one or more programs by the one or more processors 420 includes instructions for performing the following operations: performing lip region detection on a target image, acquiring a lip region image included in the target image; determining the lip region image The state of the lip region is determined according to the feature values ​​corresponding to the plurality of key feature points; the state of the lip region includes an open state or a closed state.

[0084] Further, the processor 420 is further specifically configured to execute the one or more programs including instructions for performing the following operations: according to at least one of the height value, the slope value, or the area value corresponding to the multiple key feature points A state that determines the lip area.

[0085] Further, the processor 420 is further specifically configured to execute the one or more programs including instructions for performing the following operations: determine the height difference of at least two key feature points in the plurality of key feature points; if The height difference is greater than or equal to a set height threshold, and it is determined that the state of the lip region is an open state.

[0086] Further, the processor 420 is further specifically configured to execute the one or more programs including instructions for performing the following operations: determining at least two key features in the current Nth frame of the multiple key feature points The height difference of the points is regarded as the first height difference; the height difference of the at least two key feature points in the NM image is determined as the second height difference; if the change value of the first height difference and the second height difference is greater than or equal to The set height change threshold determines that the state of the lip region is an open state; where M is a positive integer less than N.

[0087] Further, the processor 420 is specifically configured to execute the one or more programs including instructions for performing the following operations: determining the slope of a straight line formed by at least two key feature points among the plurality of key feature points Value; if the slope value is greater than or equal to the set slope threshold, it is determined that the state of the lip region is an open state.

[0088] Further, the processor 420 is further specifically configured to execute the one or more programs including instructions for performing the following operations: determining the area value of a figure formed by at least three feature points among the plurality of key feature points If the area value is greater than or equal to the set area threshold, it is determined that the state of the lip region is an open state.

[0089] Further, the processor 420 is specifically configured to execute the one or more programs including instructions for performing the following operations: use the Nth frame image as the current lip region image, and obtain the current NMth frame lip region image Wherein, M is a positive integer less than N; determine the frame difference between the current lip area image and the NM-th frame lip area image; determine the frame difference according to the feature values ​​corresponding to the multiple key feature points and the frame difference State the state of the lip area.

[0090] Further, the processor 420 is further specifically configured to execute the one or more programs including instructions for performing the following operations: extracting HOG features of the directional gradient histogram of the lip region, and determining the HOG feature according to the HOG feature The classification result of the state of the lip region; the state of the lip region is determined according to at least two of the feature values ​​corresponding to the plurality of key feature points, the classification result, and the frame difference of the lip region image.

[0091] In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as the memory 404 including instructions, which may be executed by the processor 420 of the device 400 to complete the foregoing method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0092] A machine-readable medium, for example, the machine-readable medium may be a non-transitory computer-readable storage medium. When the instructions in the medium are executed by the processor of the device (terminal or server), the device can execute such as Figure 1 to Figure 2 The lip condition detection method shown.

[0093] Figure 5 It is a schematic diagram of the structure of the server in the embodiment of the present invention. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 522 (for example, one or more processors) and a memory 532, one or one The above storage medium 530 (for example, one or one storage device with a large amount of storage) for storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 may be short-term storage or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of command operations on the server. Furthermore, the central processing unit 522 may be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the server 500.

[0094] The server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, one or more keyboards 556, and/or, one or more operating systems 541 , Such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM and so on.

[0095] After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other embodiments of the present invention. The present invention is intended to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include common knowledge or conventional technical means in the technical field not disclosed in this disclosure. . The description and the embodiments are to be regarded as exemplary only, and the true scope and spirit of the present invention are pointed out by the following claims.

[0096] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims

[0097] The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection of the present invention. Within range.

[0098] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply one of these entities or operations. There is any such actual relationship or order between. Moreover, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also includes Other elements of, or also include elements inherent to this process, method, article or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment including the element. The invention can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0099] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement it without creative work. The above are only specific embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.