Shielding object identification method and device and earphone

Abstract

The invention relates to a shielding object recognition method and device and an earphone, and the method comprises the steps: obtaining an image collected by a camera when a user wears the earphone, and enabling the camera to be disposed on the earphone; determining an image corresponding to a head area according to the image collected by the camera; cutting an image corresponding to the head area from the image acquired by the camera to obtain a cut image; and inputting the cut image into a trained deep learning model to obtain a recognition result of the shelter, the recognition result including whether the shelter exists and the category of the shelter.

Description

Technical Field

[0001] The present disclosure relates to image processing technology, and more specifically, to an obstruction recognition method, device and earphone. Background Art

[0002] Smart headphones have become a very popular electronic product on the market. They can realize audio playback and meet people's life and learning needs, especially TWS (True Wireless Stereo) headphones, which realize wireless audio transmission through Bluetooth technology, providing a convenient wearing experience and rich functions (such as listening to music, making calls, active noise reduction, spatial sound effects, etc.).

[0003] At present, the functions of smart headphones are mainly focused on audio, and other functions, such as shooting functions, are also beginning to be involved. After the user wears the headphones, the shooting angle of the camera set on the headphones may be blocked by an obstruction, affecting the use of the shooting function of the headphones.

[0004] Therefore, it is necessary to provide a new technical solution to identify obstructions.

[0005] Public Content

[0006] One objective of the present disclosure is to provide a new technical solution for an obstruction recognition method.

[0007] According to a first aspect of the present disclosure, a method for identifying an obstruction is provided, comprising:

[0008] Acquire an image captured by a camera when a user wears headphones, wherein the camera is disposed on the headphones;

[0009] Determine an image corresponding to the head area according to the image captured by the camera;

[0010] Cropping the image corresponding to the head area from the image captured by the camera to obtain a cropped image;

[0011] The cropped image is input into a trained deep learning model to obtain an occluder recognition result, wherein the recognition result includes whether there is an occluder and the type of the occluder.

[0012] Optionally, the recognition result also includes the location of the obstruction.

[0013] Optionally, the method further includes:

[0014] When the recognition result indicates that an obstruction exists, a prompt indicating the existence of the obstruction is issued to the user.

[0015] Optionally, the headset includes a speaker, wherein, when the recognition result is that an obstruction exists, prompting the user that the obstruction exists includes:

[0016] When the recognition result indicates that an obstruction exists, the speaker is controlled to emit a prompt voice to remind the user that an obstruction exists.

[0017] Optionally, the headset includes a vibration motor, wherein, when the recognition result is that an obstruction exists, prompting the user that the obstruction exists includes:

[0018] When the recognition result indicates that there is an obstruction, the vibration motor is controlled to generate vibration to prompt the user that there is an obstruction.

[0019] Optionally, determining the image corresponding to the head area according to the image captured by the camera includes:

[0020] Using a head detection algorithm, determine the face contour and / or ear contour from the image captured by the camera;

[0021] Determine an image corresponding to a face region according to the face contour, and / or determine an image corresponding to an ear region according to the ear contour;

[0022] The image corresponding to the head area is determined according to the image corresponding to the face area and / or the image corresponding to the ear area.

[0023] Optionally, the method further includes:

[0024] Obtain a training sample set and a test sample set, each sample including an image after cropping the head area and a labeling result of an occlusion, wherein the labeling result of the occlusion includes whether an occlusion exists and a category of the occlusion;

[0025] Using the training sample set to train the deep learning model to be trained to obtain a trained deep learning model;

[0026] Using the test sample set, verifying the trained deep learning model to obtain a verification result;

[0027] When the verification result is that the accuracy of the output result of the trained deep learning model reaches a preset threshold, the trained deep learning model is used as the trained deep learning model.

[0028] Optionally, before determining the image corresponding to the head area according to the image captured by the camera, the method further includes:

[0029] The image captured by the camera is subjected to denoising and contrast adjustment to obtain a processed image.

[0030] According to a second aspect of the present disclosure, there is provided a device for identifying an obstruction, comprising:

[0031] An acquisition module, used to acquire an image captured by a camera when a user wears headphones, wherein the camera is arranged on the headphones;

[0032] A head region determination module, used to determine an image corresponding to the head region based on the image captured by the camera;

[0033] An image cropping module, used to crop the image corresponding to the head area from the image captured by the camera to obtain a cropped image;

[0034] The occluder recognition module is used to input the cropped image into a trained deep learning model to obtain an occluder recognition result, wherein the recognition result includes whether an occluder exists and the type of the occluder.

[0035] According to a third aspect of the present disclosure, a headset is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of the first aspects of the present disclosure.

[0036] The occlusion recognition method provided by the present invention crops the image corresponding to the head area from the image captured by the camera, and then uses the trained deep learning model to obtain the recognition result of the occlusion. On the one hand, the recognition result of the occlusion is obtained by using the cropped image, which reduces the calculation amount of the trained deep learning model. On the other hand, the influence of the fixed occlusion of the head part is eliminated. When combined with the trained deep learning model, the accuracy of identifying temporary occlusions can be improved.

[0037] Features and advantages of the embodiments of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the embodiments of the specification.

[0039] Figure 1 is a processing flow chart of a method for identifying an obstruction according to an embodiment of the present invention.

[0040] Figure 2 4 is a principle block diagram of an obstruction recognition device according to an embodiment of the present invention.

[0041] Figure 3 FIG. 4 is a block diagram of the hardware structure of a headset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification and its application or uses.

[0044] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0045] In one embodiment of the present invention, a method for identifying an obstruction is provided. The method for identifying an obstruction is implemented by a headset. Figure 1 As shown, the obstruction recognition method of this embodiment may include the following steps S110 to S140.

[0046] Step S110, obtaining an image captured by a camera when the user wears the headset, where the camera is set on the headset.

[0047] The earphones are usually used as a pair, namely a left earphone and a right earphone.

[0048] In one embodiment, there is one camera, which is arranged on the left earphone or the right earphone. The image is a static image taken by the camera alone, or the image is a frame of image extracted from a video taken by the camera.

[0049] In one embodiment, the camera includes two cameras, namely, a first camera and a second camera. The first camera is arranged on the left earphone, and the second camera is arranged on the right earphone. The image includes a first image captured by the first camera and a second image captured by the second camera. The first image is a static image captured by the first camera alone, or the first image is a frame of image extracted from a video captured by the first camera. The second image is a static image captured by the second camera alone, or the second image is a frame of image extracted from a video captured by the second camera.

[0050] It should be noted that after the user wears the headphones, the camera set on the headphones is close to the user's head, so that the camera's shooting angle is blocked by the user's face and / or ears, and the image captured by the camera includes images corresponding to the user's face and / or images corresponding to the user's ears.

[0051] Step S120: determining an image corresponding to the head area according to the image captured by the camera.

[0052] In one embodiment, step S120 specifically includes: using a head detection algorithm to determine the face contour and / or ear contour from the image captured by the camera; determining the image corresponding to the face area based on the face contour, and / or, determining the image corresponding to the ear area based on the ear contour; determining the image corresponding to the head area based on the image corresponding to the face area and / or the image corresponding to the ear area.

[0053] The head detection algorithm can be an MTCNN (Multi-task convolutional neural network) algorithm, an SSD (Single Shot MultiBox Detector) algorithm, or the like.

[0054] In one embodiment, before executing step S120, the method further includes: performing denoising and adjusting contrast on the image captured by the camera to obtain a processed image. The processed image is used in subsequent execution steps, which can reduce useless interference information and the influence of noise in the image. In addition, adjusting the contrast can change the intensity distribution of image pixels, making the difference between the bright and dark parts of the image more obvious, which is helpful for the subsequent determination of the image of the head area.

[0055] Step S130, cropping the image corresponding to the head area from the image captured by the camera to obtain a cropped image.

[0056] The cropped image is the image captured by the camera that is not blocked by the user's head.

[0057] Step S140, input the cropped image into the trained deep learning model to obtain the recognition result of the occluder, and the recognition result includes whether the occluder exists and the category of the occluder.

[0058] If the result of the item "whether there is an obstruction" in the recognition result is yes, the result of the item "category of the obstruction" in the recognition result is a specific category. If the result of the item "whether there is an obstruction" in the recognition result is no, the result of the item "category of the obstruction" in the recognition result is none.

[0059] In one embodiment, the training process of the trained deep learning model is as follows: obtain a training sample set and a test sample set, each sample includes an image after cropping the head area and the annotation result of the occlusion, and the annotation result of the occlusion includes whether there is an occlusion and the category of the occlusion; use the training sample set to train the deep learning model to obtain the trained deep learning model; use the test sample set to verify the trained deep learning model to obtain the verification result; when the verification result is that the accuracy of the output result of the trained deep learning model reaches a preset threshold, the trained deep learning model is used as the trained deep learning model. The training process of the deep learning model is the optimization process of the parameters in the deep learning model. The image after cropping the head area in each sample is cropped based on the image captured by the camera set on the headset. The specific method of determining the head area can refer to the above embodiment, and no further details are given here.

[0060] When the verification result is that the accuracy of the output result of the trained deep learning model does not reach the preset threshold, continue to train the deep learning model using the training sample set until the accuracy of the output result of the trained deep learning model reaches the preset threshold using the test sample set, and stop training the deep learning model.

[0061] The labeling results of occlusions are based on manual labeling. The labeling results of occlusions can be hair, glasses, hands, hats, etc.

[0062] The deep learning model can be a MobileNet model, a YOLO model, or an SSD (Single Shot MultiBoxDetector) model. The MobileNet model uses a Depthwise Separable Convolution algorithm.

[0063] In one embodiment, the recognition result further includes the position of the obstruction. The position of the obstruction is a position relative to the headset. For example, the position of the obstruction is located in front of the headset camera, upper left, upper right, lower left or lower right.

[0064] In one embodiment, when there are two cameras, namely, a first camera disposed on the left earphone and a second camera disposed on the right earphone, firstly, based on the first image captured by the first camera, it is determined whether there is an obstruction in the shooting angle of the first camera and the type of the obstruction, and based on the second image captured by the second camera, it is determined whether there is an obstruction in the shooting angle of the second camera and the type of the obstruction. When it is determined that there is an obstruction in the shooting angle of the first camera, the position of the obstruction is determined, and when it is determined that there is an obstruction in the shooting angle of the second camera, the position of the obstruction is determined.

[0065] In one embodiment, the method further includes: when the recognition result is that there is an obstruction, a prompt is issued to the user that there is an obstruction. In this way, when there is a temporary obstruction, the user can be promptly prompted so that the user can promptly eliminate the influence of the obstruction on the camera shooting.

[0066] In one embodiment, the earphone includes a speaker. When the recognition result shows that an obstruction exists, the speaker is controlled to emit a prompt voice to remind the user that an obstruction exists.

[0067] In one embodiment, the headset includes a vibration motor. When the recognition result shows that an obstruction exists, the vibration motor is controlled to vibrate to prompt the user that the obstruction exists.

[0068] The method for identifying occluders provided by the present invention first crops the image corresponding to the head area from the image captured by the camera, and then uses the trained deep learning model to obtain the identification result of the occluder. On the one hand, the cropped image is used to obtain the identification result of the occluder, which reduces the calculation amount of the trained deep learning model. On the other hand, the influence of the fixed occlusion of the face part is eliminated. When combined with the trained deep learning model, the accuracy of identifying temporary occluders can be improved.

[0069] The present invention also provides an obstruction recognition device for implementing any of the above method embodiments. Figure 2 FIG. 2 shows a structural block diagram of an obstruction recognition device according to an embodiment of the present invention. Figure 2 As shown, the obstruction recognition device includes an external sound acquisition module 210 , a head region determination module 220 , an image cropping module 230 and an obstruction recognition module 240 .

[0070] The acquisition module 210 is used to acquire images captured by a camera when a user wears headphones, and the camera is set on the headphones.

[0071] The head region determination module 220 is used to determine an image corresponding to the head region according to the image captured by the camera.

[0072] The image cropping module 230 is used to crop the image corresponding to the head area from the image captured by the camera to obtain a cropped image.

[0073] The occluder recognition module 240 is used to input the cropped image into a trained deep learning model to obtain an occluder recognition result, where the recognition result includes whether an occluder exists and the type of the occluder.

[0074] In one embodiment, the recognition result of the occluder also includes the position of the occluder.

[0075] In one embodiment, the device further comprises a prompt module, which is used to issue a prompt to the user that an obstruction exists when the recognition result indicates that an obstruction exists.

[0076] In one embodiment, the prompt module is used to control the speaker to emit a prompt voice to remind the user that there is an obstruction when the recognition result is that there is an obstruction.

[0077] In one embodiment, the prompt module is used to control the vibration motor to generate vibration when the recognition result is that there is an obstruction, so as to prompt the user that there is an obstruction.

[0078] In one embodiment, the head region determination module is used to determine the face contour from the image captured by the camera using a face detection algorithm; and determine the image corresponding to the head region based on the face contour.

[0079] In one embodiment, the device further includes a training module. The training module is used to obtain a training sample set and a test sample set, each sample including an image after cropping the head area and an annotation result of an occlusion, the annotation result of the occlusion including whether there is an occlusion and the category of the occlusion; the deep learning model to be trained is trained using the training sample set to obtain a trained deep learning model; the trained deep learning model is verified using the test sample set to obtain a verification result; when the verification result is that the accuracy of the output result of the trained deep learning model reaches a preset threshold, the trained deep learning model is used as a trained deep learning model.

[0080] The present invention also provides a headset for implementing any of the above method embodiments. Figure 3 FIG. 4 shows a hardware structure block diagram of a headset 300 according to an embodiment of the present invention. Figure 3 As shown, the headset 300 includes a processor 310 and a memory 320 for storing instructions executable by the processor 310. The processor 310 is configured to implement the obstruction recognition method according to any embodiment of the present disclosure when executing the instructions stored in the memory 320.

[0081] The processor 310 is used to execute computer instructions, which can be written in an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 320 includes, for example, ROM (read-only memory), RAM (random access memory), a non-volatile memory such as a hard disk, etc., which are not limited here.

[0082] In this embodiment, the headset further includes a camera, and the camera is used to send images captured by the camera to the processor.

[0083] In one embodiment, the headset further comprises a speaker. The processor can also be used to control the speaker to emit a prompt voice to remind the user that there is an obstruction when the recognition result shows that there is an obstruction.

[0084] In one embodiment, the headset further comprises a vibration motor. The processor can also be used to control the vibration motor to vibrate when the recognition result shows that an obstruction exists, so as to prompt the user that an obstruction exists.

[0085] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, its related parts can be referred to the partial description of the method embodiment.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] The embodiments of the present specification may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer instructions for causing a processor to implement various aspects of the embodiments of the present specification.

[0088] A computer-readable storage medium may be a tangible device that can hold and store computer instructions used by a computer instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which computer instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0089] The computer instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network layer, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network layer may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network layer adapter card or network layer interface in each computing / processing device receives the computer instructions from the network layer and forwards the computer instructions for storage in a computer-readable storage medium in each computing / processing device.

[0090] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a computer instruction, and a part of a module, a program segment or a computer instruction contains one or more executable computer instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0091] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying an obstruction, characterized in that: include: Acquire an image captured by a camera when a user wears headphones, wherein the camera is disposed on the headphones; Determine an image corresponding to the head area according to the image captured by the camera; Cropping the image corresponding to the head area from the image captured by the camera to obtain a cropped image; The cropped image is input into a trained deep learning model to obtain an occluder recognition result, wherein the recognition result includes whether there is an occluder and the type of the occluder.

2. The method according to claim 1, characterized in that The recognition result also includes the position of the blocking object.

3. The method according to claim 1, characterized in that The method further comprises: When the recognition result indicates that an obstruction exists, a prompt indicating the existence of the obstruction is issued to the user.

4. The method according to claim 3, characterized in that The earphone includes a speaker, wherein, when the recognition result is that an obstruction exists, prompting the user that the obstruction exists includes: When the recognition result indicates that an obstruction exists, the speaker is controlled to emit a prompt voice to remind the user that an obstruction exists.

5. The method according to claim 3, characterized in that: The headset includes a vibration motor, wherein, when the recognition result is that an obstruction exists, prompting the user that the obstruction exists includes: When the recognition result indicates that there is an obstruction, the vibration motor is controlled to generate vibration to prompt the user that there is an obstruction.

6. The method according to claim 1, characterized in that The step of determining an image corresponding to the head region according to the image captured by the camera includes: Using a head detection algorithm, determine the face contour and / or ear contour from the image captured by the camera; Determine an image corresponding to a face region according to the face contour, and / or determine an image corresponding to an ear region according to the ear contour; The image corresponding to the head area is determined according to the image corresponding to the face area and / or the image corresponding to the ear area.

7. The method according to claim 1, characterized in that The method further comprises: Obtain a training sample set and a test sample set, each sample including an image after cropping the head area and a labeling result of an occlusion, wherein the labeling result of the occlusion includes whether an occlusion exists and a category of the occlusion; Using the training sample set to train the deep learning model to be trained to obtain a trained deep learning model; Using the test sample set, verifying the trained deep learning model to obtain a verification result; When the verification result is that the accuracy of the output result of the trained deep learning model reaches a preset threshold, the trained deep learning model is used as the trained deep learning model.

8. The method according to any one of claims 1 to 7, characterized in that: Before determining the image corresponding to the head area according to the image captured by the camera, the method further includes: The image captured by the camera is subjected to denoising and contrast adjustment to obtain a processed image.

9. An obstruction recognition device, characterized in that: include: An acquisition module, used to acquire an image captured by a camera when a user wears headphones, wherein the camera is arranged on the headphones; A head region determination module, used to determine an image corresponding to the head region based on the image captured by the camera; An image cropping module, used to crop the image corresponding to the head area from the image captured by the camera to obtain a cropped image; The occluder recognition module is used to input the cropped image into a trained deep learning model to obtain an occluder recognition result, wherein the recognition result includes whether an occluder exists and the type of the occluder.

10. A headset, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of claims 1 to 8.

Images