Interaction method and device based on earphone camera module, equipment and storage medium

Abstract

The invention relates to the field of earphones, in particular to an interaction method and device based on an earphone camera module, equipment and a storage medium. The earphone comprises a first earphone main body and a second earphone main body, the first earphone main body and the second earphone main body are respectively provided with a camera module, and the method comprises the following steps: obtaining a first image and a second image, the first image and the second image being images shot by the camera modules of the first earphone main body and the second earphone main body at the same time; determining an overlapping area of the first image and the second image; and playing voice information related to the target object through the first earphone main body and the second earphone main body under the condition that the overlapping area comprises the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of headphones, and more specifically, to an interaction method, device, equipment and storage medium based on a headphone camera module. Background Art

[0002] With the rapid development of science and technology, smart wearable devices have become an indispensable part of people's daily lives. As one of the most common smart wearable devices, the functions of headphones are no longer limited to audio playback and calls. More and more innovative technologies are being integrated into headphones to meet the diverse needs of users. At present, in some scenarios, such as museums, the sensing accuracy of explanation headphones is limited, resulting in insufficient sensitivity and low accuracy in explanation recognition, which may cause tourists to miss important information points or encounter false triggers during the visit, affecting the tour experience. Summary of the invention

[0003] One purpose of the present disclosure is to provide an interactive method based on an earphone camera module, which can achieve a more accurate explanation function through the earphone camera module.

[0004] According to a first aspect of the present disclosure, an interaction method based on an earphone camera module is provided, wherein the earphone comprises a first earphone body and a second earphone body, wherein the first earphone body and the second earphone body are respectively provided with a camera module, and the method comprises:

[0005] Acquire a first image and a second image, wherein the first image and the second image are images taken by camera modules of the first earphone body and the second earphone body at the same time;

[0006] determining an overlapping area between the first image and the second image;

[0007] In a case where the overlapping area includes a target object, voice information related to the target object is played through the first earphone body and the second earphone body.

[0008] Optionally, determining an overlapping area between the first image and the second image includes:

[0009] Detecting feature points in the first image and the second image, and matching the feature points;

[0010] Calculate the transformation matrix between the two images through the screened matching point pairs;

[0011] The vertices and feature points of the first image and the second image are mapped into the same coordinate system, and the overlapping area of ​​the first image and the second image is determined.

[0012] Optionally, after playing the voice information related to the target object through the first earphone body and the second earphone body, the method further includes:

[0013] Based on a preset time interval, continuously acquiring the first image and the second image;

[0014] Determining a position of the target object in the first image and the second image;

[0015] When the target object is not in any area of ​​the first image and the second image, the voice information related to the target object is stopped from being played.

[0016] Optionally, the first earphone body and the second earphone body are respectively provided with an inertial measurement unit, and after the voice information related to the target object is played through the first earphone body and the second earphone body, the method further includes:

[0017] Based on a preset time interval, continuously acquiring the first image and the second image;

[0018] Determining a position of the target object in the first image and the second image;

[0019] Acquire a movement distance between the first earphone body and the second earphone body;

[0020] When the target object is not in any area of ​​the first image and the second image, and the movement distances of the first earphone body and the second earphone body are greater than a speed threshold, the voice information related to the target object is stopped from being played.

[0021] Optionally, the first earphone body and the second earphone body are respectively provided with an inertial measurement unit, and after the voice information related to the target object is played through the first earphone body and the second earphone body, the method further includes:

[0022] Based on a preset time interval, continuously acquiring the first image and the second image;

[0023] Determining a position of the target object in the first image and the second image;

[0024] Acquire the movement speed and movement direction of the first earphone body and the second earphone body;

[0025] When the angle between the movement directions of the first earphone body and the second earphone body is smaller than a preset angle threshold, the movement speeds of the first earphone body and the second earphone body are greater than a preset speed threshold, and the target object is not in any area of ​​the first image and the second image, stop playing the voice information related to the target object.

[0026] Optionally, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and playing voice information related to the target objects through the first earphone body and the second earphone body includes:

[0027] Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects;

[0028] Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether a target gesture exists in the first image and the second image;

[0029] When the target gesture appears in the first area, it is determined to play the relevant voice information of the first target object; when the target gesture appears in the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

[0030] Optionally, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and playing voice information related to the target objects through the first earphone body and the second earphone body includes:

[0031] Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects;

[0032] Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether a target gesture exists in the first image and the second image;

[0033] When the user's hand moves from the overlapping area to the first area, it is determined to play the relevant voice information of the first target object; when the user's hand moves from the overlapping area to the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

[0034] According to a second aspect of the present disclosure, an interactive device based on an earphone camera module is provided, wherein the earphone comprises a first earphone body and a second earphone body, wherein the first earphone body and the second earphone body are respectively provided with a camera module, and the device comprises:

[0035] An acquisition module, configured to acquire a first image and a second image, wherein the first image and the second image are images taken by the first earphone body and the second earphone body at the same time respectively;

[0036] A determination module, used to determine an overlapping area between the first image and the second image;

[0037] The playing module is used for playing the voice information related to the target object through the first earphone body and the second earphone body when the overlapping area includes the target object.

[0038] According to a third aspect of the present disclosure, a hearing electronic device is provided, comprising a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the steps of any one of the methods described in the first aspect are implemented.

[0039] According to a fourth aspect of the present disclosure, there is provided a storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0040] One technical effect of the present disclosure is that an interaction method based on an earphone camera module is provided, wherein the earphone includes a first earphone body and a second earphone body, and a first image and a second image can be obtained, wherein the first image and the second image are respectively images taken by the first earphone body and the second earphone body at the same time; and whether the overlapping area of ​​the two images includes a target object is detected, so as to play voice information related to the target object. In this way, the user's attention range can be determined based on the overlapping area of ​​the two images, and when the user pays attention to the target object, relevant information of the corresponding object, such as an item introduction, can be played.

[0041] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0043] Figure 1 is a flow chart of an interaction method based on a headset camera module according to an embodiment;

[0044] Figure 2 is a schematic diagram of an overlap region according to one embodiment;

[0045] Figure 3 is a schematic diagram of a first image according to an embodiment;

[0046] Figure 4 is a schematic diagram of a second image according to an embodiment;

[0047] Figure 5 It is a structural schematic diagram of an interactive device based on an earphone camera module according to an embodiment. DETAILED DESCRIPTION

[0048] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0050] Techniques and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the techniques and equipment should be considered part of the specification.

[0051] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0052] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0053] The embodiment of the present application provides an interaction method of an earphone camera module, which can be applied to an earphone or an electronic device connected to the earphone. The earphone includes a first earphone body and a second earphone body, and the first earphone body and the second earphone body are respectively provided with a camera module. The method includes steps S11-S13.

[0054] Step S11, acquiring a first image and a second image, wherein the first image and the second image are images taken by camera modules of the first earphone body and the second earphone body at the same time respectively.

[0055] In this example, the earphones can be TWS (True Wireless Stereo, true wireless stereo earphones) earphones, or they can be headphones or open earphones, audio glasses. Taking TWS earphones as an example, it can include two earphone bodies suitable for the left and right ears of the user, and a charging storage device. In an embodiment of the present application, the camera module of the earphone can be set on both earphone bodies, and the processing device can be set on the earphone body or in the charging storage device. In the headphone system, the camera module can be set in a group near the left and right ears, respectively, to detect heart rate data respectively.

[0056] In the embodiment of the present application, the earphone body can be a small earplug or a comfortable ear hook or ear clip worn on the left and right sides of a wireless earphone; at the same time, it can also be a large earmuff in a head-mounted headset that wraps around the ears and provides excellent sound insulation, or the main part of an all-in-one earphone that integrates multiple functions and also covers the ears.

[0057] In this embodiment, the camera modules of the two earphone bodies can be controlled to start shooting, and two images shot by the two earphone bodies at the same time can be obtained respectively.

[0058] Step S12: determining an overlapping area between the first image and the second image.

[0059] In this embodiment, determining the overlapping area of ​​the first image and the second image includes: detecting feature points in the first image and the second image, and matching the feature points; calculating the transformation matrix between the two images through the screened matching point pairs; mapping the vertices and feature points of the first image and the second image to the same coordinate system, and determining the overlapping area of ​​the first image and the second image.

[0060] In this example, feature point detection algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF) can be used to detect and describe the feature points of the first image and the second image, and generate a feature point set and a corresponding descriptor. After that, the feature points of the two images are matched by a feature matching algorithm to obtain matching feature point pairs. After that, the matching results can be filtered to remove mismatched points and improve the matching accuracy. Then, the RANSAC (Random Sample Consensus) algorithm or other robust algorithms are used to calculate the homography matrix, that is, to map the feature points to the same coordinate system. For example, 4 point pairs can be randomly selected from the screened matching point pairs, and the homography matrix can be calculated using these 4 point pairs. The homography matrix is ​​a 3x3 matrix that describes the projection transformation from one image plane to another image plane. Use the calculated homography matrix to perform error evaluation on all remaining matching point pairs, that is, calculate the reprojection error of each point pair after transformation. Point pairs with a reprojection error less than a certain threshold (such as 5.0 pixels) are considered inliers, that is, these point pairs conform to the current homography matrix model. If the number of inliers is large enough, recalculate the homography matrix based on all inliers and repeat the above process. Iterate a certain number of times or stop until a certain convergence condition is met. Using the homography matrix, map the vertices or key feature points of one image to the coordinate system of the other image, and determine the overlapping area by calculating the intersection of the mapped points and the edge of the image.

[0061] In this embodiment, since the position of the earphone when worn is relatively symmetrical, such as Figure 2 As shown, the overlapping area that can be photographed by the two headphone bodies is usually the area directly in front of the user's head. Therefore, the area where the user's attention is focused can be determined based on the overlapping area.

[0062] Step S13: when the overlapping area includes the target object, voice information related to the target object is played through the first earphone body and the second earphone body.

[0063] In this embodiment, the target object in the image can be identified by an image recognition algorithm. The image recognition algorithm can be a pre-trained YOLO algorithm or an algorithm such as SSD, and the target object can be a pre-stored or set object. For example, in a museum scene, the target object can be a cultural relic, a signboard or other object exhibited in the venue. In this embodiment, the headset can communicate with other devices such as a museum or a server based on wireless communication, such as Bluetooth or WIFI, and obtain relevant voice information of the target object, as well as image feature information, etc. through the wireless communication.

[0064] In this embodiment, when the target object is in the overlapping area, it can be said that the user's attention is also on the target object. At this time, voice information related to the target object can be played through the earphone, such as introduction information of museum exhibits, price information of goods, etc.

[0065] In this example, an interaction method based on an earphone camera module is provided, wherein the earphone includes a first earphone body and a second earphone body, and a first image and a second image can be acquired, wherein the first image and the second image are respectively images taken by the first earphone body and the second earphone body at the same time; and it is detected whether the overlapping area of ​​the two images includes a target object, so as to play voice information related to the target object. In this way, the user's attention range can be determined based on the overlapping area of ​​the two images, and when the user pays attention to the target object, the relevant information of the corresponding object can be played.

[0066] In an example of this embodiment, after playing voice information related to the target object through the first earphone body and the second earphone body, the method also includes: continuously acquiring the first image and the second image based on a preset time interval, and determining the position of the target object in the first image and the second image; when the target object is not in any area of ​​the first image and the second image, stopping playing the voice information related to the target object.

[0067] In this embodiment, after playing the voice information, the camera can continuously capture images through the camera modules of the two earphone bodies at a preset time interval or frequency, and continuously determine the position of the target object in the image. Generally speaking, when the user pays attention to the target object, he or she may turn his or her head, and in this case the target object usually does not completely leave the range of the first image and the second image. Therefore, when the target object leaves the overlapping area but does not leave other areas, the relevant voice information can be kept playing. Only when the user completely loses attention to it and no longer pays attention to it, will the target object completely leave the first image and the second image. At this time, the voice information related to the target object can be stopped from being played.

[0068] In this example, when the target object does not appear in any area of ​​the first image or the second image, it can be determined that the user has lost attention on the target object, and the relevant information is stopped from being played, so as to save power consumption of the headset and improve the user experience.

[0069] In an example of the present embodiment, inertial measurement units are respectively provided on the first earphone body and the second earphone body. After the voice information related to the target object is played through the first earphone body and the second earphone body, the method further includes: continuously acquiring the first image and the second image based on a preset time interval; determining the position of the target object in the first image and the second image; acquiring motion parameters of the first earphone body and the second earphone body; and stopping playing the voice information related to the target object when the target object is not in any area of ​​the first image and the second image and the movement distance of the first earphone body and the second earphone body is greater than a speed threshold.

[0070] In this embodiment, after playing the voice information, the camera can continuously capture images through the camera modules of the two headset bodies at a preset time interval or frequency, and continuously determine the position of the target object in the image. Generally speaking, when the user pays attention to the target object, he may turn his head, etc. In this case, the target object usually does not completely leave the range of the first image and the second image. However, it cannot completely avoid the user being suddenly affected by other factors around the target object, thereby turning around, so that the target object completely leaves the range of the first image and the second image. Therefore, the movement distance of the headset body can be obtained based on the inertial measurement unit in the headset body. The specific inertial measurement unit can integrate the detected acceleration data to obtain the movement distance. When the movement distance is greater than the preset threshold, it can be determined that the user has completely lost attention to the target object, and the voice information related to the target object can be stopped. When the distance is not greater than the threshold, the voice information related to the target object can be played without stopping.

[0071] In an example of the present embodiment, inertial measurement units are respectively provided on the first earphone body and the second earphone body. After playing voice information related to the target object through the first earphone body and the second earphone body, the method also includes: continuously acquiring the first image and the second image based on a preset time interval; determining the position of the target object in the first image and the second image; acquiring the movement speed and movement direction of the first earphone body and the second earphone body; and stopping playing the voice information related to the target object when the angle between the movement directions of the first earphone body and the second earphone body is less than a preset angle threshold, the movement speeds of the first earphone body and the second earphone body are greater than a preset speed threshold, and the target object is not in any area of ​​the first image and the second image.

[0072] In this embodiment, after playing the voice information, the camera can continuously capture images through the camera modules of the two earphone bodies at a preset time interval or frequency, and continuously determine the position of the target object in the image. In some cases, the movement speed of the earphone cannot accurately determine the change of the user's attention. At this time, the movement direction of the earphone can be combined to determine whether the user is still paying attention to the target object. In this example, the preset angle threshold can be set to a value less than 180 degrees. Due to the characteristics of the human body structure, when the user turns his head, the movement directions of the two earphones are opposite. Therefore, according to actual needs, the preset angle threshold can be set to any value of 180 degrees, such as 30 degrees, 45 degrees, or even 120 degrees. Therefore, when the movement speed of the two earphone bodies is greater than the speed threshold, the movement direction is less than the angle threshold, and the target object does not appear in any area of ​​the first image and the second image, it means that the user is moving away from the target object quickly and the attention is not on the target object. At this time, the voice information related to the target object can be stopped. When the movement speeds of the two earphone bodies are greater than the speed threshold, the movement directions are greater than the angle threshold, and the target object does not appear in any area of ​​the first image and the second image, it means that the user only turned his head significantly and his body did not move significantly. At this time, the relevant voice information of the target object can continue to be played.

[0073] In an example of the present embodiment, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and voice information related to the target objects is played through the first earphone body and the second earphone body, including: determining the first target object and the second target object, wherein the first target object is the target object relatively positioned to the left of the two target objects, and the second target object is the target object relatively positioned to the right of the two target objects; continuing to acquire the first image and the second image, and determining whether there is a target gesture in the first image and the second image; when the target gesture appears in the first area, determining to play the relevant voice information of the first target object, and when the target gesture appears in the second area, determining to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

[0074] In this embodiment, two or more target objects may appear in the overlapping area of ​​the first earphone body and the second earphone body. In the case of two target objects, the relative positions of the two target objects can be determined first to distinguish the target objects on the left and the target objects on the right. Afterwards, it can be determined whether the target gesture appears in the first area and the second area in the images captured by the two earphone camera modules. Figure 3 As shown, the area of ​​the second image is Figure 4As shown, the right area of ​​the first image and the left area of ​​the second image are overlapping areas, that is, the area in the direction facing the user, and the other areas are the first area and the second area. In this example, the gesture for determining the target object can be pre-set, and the image area where the gesture appears can be detected. If the user gesture appears in the first area, that is, the area on the left side of the user, then the relevant voice information of the first target object on the left can be played. If the user gesture appears in the second area, that is, the area on the right side of the user, then the relevant voice information of the second target object on the right can be played.

[0075] In another example, if multiple target objects appear, the relative positions of the multiple target objects can be determined and their left and right positions can be distinguished. After that, the voice information of the target object in the middle position among the multiple target objects can be played first, and then the images captured by the two earphone camera modules can be continuously acquired to determine whether the target gesture appears in the first area and the second area. When the target gesture appears in the first area, the voice information of the target object being played can be switched to the voice information of the target object on the left. When the target gesture appears in the second area, the voice information of the target object being played can be switched to the voice information of the target object on the right.

[0076] In this embodiment, a method is provided, which can provide a user with a control method for determining a target object for which voice information is to be played when multiple target objects appear in an overlapping area, thereby improving the user's experience.

[0077] In an example of the present embodiment, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and voice information related to the target objects is played through the first earphone body and the second earphone body, including: determining the first target object and the second target object, wherein the first target object is the target object relatively positioned to the left of the two target objects, and the second target object is the target object relatively positioned to the right of the two target objects; based on a preset time interval, continuously acquiring the first image and the second image, and determining whether there is a target gesture in the first image and the second image; when the user's hand moves from the overlapping area to the first area, determining to play the relevant voice information of the first target object, and when the user's hand moves from the overlapping area to the second area, determining to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

[0078] In this embodiment, two or more target objects may appear in the overlapping area of ​​the first earphone body and the second earphone body. In the case of two target objects, the relative positions of the two target objects can be determined first, and the target objects on the left and the target objects on the right can be distinguished. Then, the specific position of the user's hand can be determined from the images captured by the two earphone camera modules. Figure 3 When the overlapping area shown in FIG. 1 moves to the first area, that is, the user's hand moves from the middle to the left, the relevant voice information of the first target object on the left can be played. Figure 4 When the overlapping area shown moves to the second area, that is, the user's hand moves from the middle to the right, the relevant voice information of the second target object on the right can be played.

[0079] In another example, if multiple target objects appear, the relative positions of the multiple target objects can be determined, and their left and right positions can be distinguished. After that, the voice information of the target object in the middle position among the multiple target objects can be played first, and then the images taken by the two earphone camera modules are continuously acquired to determine the specific position of the user's hand. When the user's hand moves from the overlapping area to the first area, that is, the user's hand moves from the middle to the left, the voice information of the target object being played is switched to the voice information of the target object on the left. When the user's hand moves from the overlapping area to the second area, that is, the user's hand moves from the middle to the right, the voice information of the target object being played can be switched to the voice information of the target object on the right.

[0080] In this embodiment, a method is provided, which can provide a user with a control method for determining a target object for which voice information is to be played when multiple target objects appear in an overlapping area, thereby improving the user's experience.

[0081] like Figure 5 As shown, the embodiment of the present application also provides an interactive device 100 based on an earphone camera module, the earphone includes a first earphone body and a second earphone body, the first earphone body and the second earphone body are respectively provided with a camera module, the device includes: an acquisition module 101, used to acquire a first image and a second image, wherein the first image and the second image are respectively images taken by the first earphone body and the second earphone body at the same time; a determination module 102, used to determine an overlapping area of ​​the first image and the second image; a playback module 103, used to play voice information related to the target object through the first earphone body and the second earphone body when the overlapping area includes the target object.

[0082] Optionally, the determination module is specifically used for:

[0083] Detecting feature points in the first image and the second image, and matching the feature points;

[0084] Calculate the transformation matrix between the two images through the screened matching point pairs;

[0085] The vertices and feature points of the first image and the second image are mapped into the same coordinate system, and the overlapping area of ​​the first image and the second image is determined.

[0086] Optionally, the acquisition module is also used to continuously acquire the first image and the second image based on a preset time interval; the determination module is also used to determine the position of the target object in the first image and the second image; the device also includes: a stop module, which is used to stop playing voice information related to the target object when the target object is not in any area of ​​the first image and the second image.

[0087] Optionally, an inertial measurement unit is respectively provided on the first earphone body and the second earphone body, for obtaining the movement distance of the first earphone body and the second earphone body; the acquisition module is also used to continuously obtain the first image and the second image based on a preset time interval; the determination module is also used to determine the position of the target object in the first image and the second image; the device also includes: a stop module, which is used to stop playing voice information related to the target object when the target object is not in any area of ​​the first image and the second image, and the movement distance of the first earphone body and the second earphone body are both greater than a speed threshold.

[0088] Optionally, the first earphone body and the second earphone body are respectively provided with an inertial measurement unit for acquiring the movement speed and movement direction of the first earphone body and the second earphone body;

[0089] The acquisition module is also used to continuously acquire the first image and the second image based on a preset time interval; the determination module is also used to determine the position of the target object in the first image and the second image; the device also includes: a stop module, which is used to stop playing the voice information related to the target object when the angle between the movement directions of the first earphone body and the second earphone body is less than a preset angle threshold, the movement speed of the first earphone body and the second earphone body is greater than a preset speed threshold, and the target object is not in any area of ​​the first image and the second image.

[0090] Optionally, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and the playback module is specifically used for:

[0091] Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects;

[0092] Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether there is a target gesture in the first image and the second image;

[0093] When the target gesture appears in the first area, it is determined to play the relevant voice information of the first target object; when the target gesture appears in the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image and the second area is a non-overlapping area in the second image.

[0094] Optionally, the first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and the playback module is specifically used for:

[0095] Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects;

[0096] Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether there is a target gesture in the first image and the second image;

[0097] When the user's hand moves from the overlapping area to the first area, it is determined to play the relevant voice information of the first target object; when the user's hand moves from the overlapping area to the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

[0098] An embodiment of the present application also provides a hearing electronic device, including a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, any one of the above-mentioned embodiments of the interaction method based on the earphone camera module is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0099] An embodiment of the present application also provides a storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, any one of the above-mentioned embodiments of the interaction method based on the headset camera module is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0100] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0101] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] The embodiments of the present disclosure may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the embodiments of the present disclosure.

[0103] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0104] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0105] The computer program instructions for performing the operation of the embodiments of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the embodiments of the present disclosure.

[0106] Various aspects of the embodiments of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and a part of a module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0110] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An interactive method based on a headset camera module, characterized in that: The earphone comprises a first earphone body and a second earphone body, wherein the first earphone body and the second earphone body are respectively provided with a camera module, and the method comprises: Acquire a first image and a second image, wherein the first image and the second image are images taken by camera modules of the first earphone body and the second earphone body at the same time; determining an overlapping area between the first image and the second image; In a case where the overlapping area includes a target object, voice information related to the target object is played through the first earphone body and the second earphone body.

2. The method according to claim 1, characterized in that The determining an overlapping area between the first image and the second image includes: Detecting feature points in the first image and the second image, and matching the feature points; Calculate the transformation matrix between the two images through the screened matching point pairs; The vertices and feature points of the first image and the second image are mapped into the same coordinate system, and the overlapping area of ​​the first image and the second image is determined.

3. The method according to claim 1, characterized in that After playing the voice information related to the target object through the first earphone body and the second earphone body, the method further includes: Based on a preset time interval, continuously acquiring the first image and the second image; Determining a position of the target object in the first image and the second image; When the target object is not in any area of ​​the first image and the second image, the voice information related to the target object is stopped from being played.

4. The method according to claim 1, characterized in that: The first earphone body and the second earphone body are respectively provided with an inertial measurement unit. After the voice information related to the target object is played through the first earphone body and the second earphone body, the method further includes: Based on a preset time interval, continuously acquiring the first image and the second image; Determining a position of the target object in the first image and the second image; Acquire a movement distance between the first earphone body and the second earphone body; When the target object is not in any area of ​​the first image and the second image, and the movement distances of the first earphone body and the second earphone body are greater than a speed threshold, the voice information related to the target object is stopped from being played.

5. The method according to claim 1, characterized in that The first earphone body and the second earphone body are respectively provided with an inertial measurement unit. After the voice information related to the target object is played through the first earphone body and the second earphone body, the method further includes: Based on a preset time interval, continuously acquiring the first image and the second image; Determining a position of the target object in the first image and the second image; Acquire the movement speed and movement direction of the first earphone body and the second earphone body; When the angle between the movement directions of the first earphone body and the second earphone body is smaller than a preset angle threshold, the movement speeds of the first earphone body and the second earphone body are greater than a preset speed threshold, and the target object is not in any area of ​​the first image and the second image, stop playing the voice information related to the target object.

6. The method according to claim 1, characterized in that The first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and the voice information related to the target objects is played through the first earphone body and the second earphone body, including: Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects; Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether a target gesture exists in the first image and the second image; When the target gesture appears in the first area, it is determined to play the relevant voice information of the first target object; when the target gesture appears in the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

7. The method according to claim 6, characterized in that The first earphone body is worn on the left ear of the user, the second earphone body is worn on the right ear of the user, the target object includes two target objects, and the voice information related to the target objects is played through the first earphone body and the second earphone body, including: Determine a first target object and a second target object, wherein the first target object is the target object located relatively to the left of the two target objects, and the second target object is the target object located relatively to the right of the two target objects; Based on a preset time interval, continuously acquiring the first image and the second image, and determining whether a target gesture exists in the first image and the second image; When the user's hand moves from the overlapping area to the first area, it is determined to play the relevant voice information of the first target object; when the user's hand moves from the overlapping area to the second area, it is determined to play the relevant voice information of the second target object, wherein the first area is a non-overlapping area in the first image, and the second area is a non-overlapping area in the second image.

8. An interactive device based on a headset camera module, characterized in that: The earphone comprises a first earphone body and a second earphone body, wherein the first earphone body and the second earphone body are respectively provided with a camera module, and the device comprises: An acquisition module, configured to acquire a first image and a second image, wherein the first image and the second image are images taken by the first earphone body and the second earphone body at the same time respectively; A determination module, used to determine an overlapping area between the first image and the second image; The playing module is used for playing the voice information related to the target object through the first earphone body and the second earphone body when the overlapping area includes the target object.

9. A hearing electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images