Head-mounted smart device, streaming media processing method, storage medium and program product

Abstract

The embodiment of the invention provides a head-mounted intelligent device, a streaming media processing method, a storage medium and a program product. In the embodiment of the invention, the head-mounted end side AI equipment comprises two head band parts and earmuff parts, and a circuit board comprising a processing chip is arranged on one head band part and is responsible for providing a model reasoning capability; the display screen is arranged on the other head band part, and the head band part provided with the display screen can move towards the face direction of the user so as to drive the display screen to move into the view field range of the user, so that the display problem of an AI picture is solved; the audio signal related to the target application is picked up or output through the earmuff part, and the perception problem of the AI is solved. Besides, the head-mounted intelligent equipment meets the requirements of the AI on the perception capability, the reasoning capability and the display capability, is different from the traditional end-side AI equipment in implementation form, is novel end-side AI equipment, and enriches the implementation form of the end-side AI equipment. Meanwhile, the head-mounted end-side AI equipment has the advantages of being small and exquisite, light, comfortable to wear and the like.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a head-mounted intelligent device, a streaming media processing method, a storage medium, and a program product. Background Art

[0002] Artificial Intelligence (AI) refers to machines or software with a certain degree of intelligence created by humans, which can understand and learn human behavior and perform tasks through self-learning and improvement. Its goal is to solve complex problems by simulating, extending and enhancing human intelligent behavior, such as knowledge representation, planning, natural language processing, perception, pattern recognition, etc.

[0003] At present, AI is mainly used in the cloud, referred to as cloud AI, which refers to the application mode in which the computing, storage and models involved in AI are deployed on cloud servers for processing. In this architecture, the user's terminal device (such as smartphones, IoT devices) collects data and sends it to the cloud server; the cloud server uses complex machine learning algorithms and deep learning models for analysis, reasoning and decision-making, and the processing results are transmitted back to the user's terminal device through the network, thereby realizing intelligent services.

[0004] With the continuous development of terminal technology, AI has gradually shifted to the end-side, that is, the computing, storage, and models involved in AI are deployed on the user's terminal devices (such as smartphones and IoT devices), which is referred to as end-side AI. End-side AI currently mainly relies on devices such as smartphones and IoT devices. With the development of AI towards the end-side, more forms of end-side AI devices are needed. Summary of the invention

[0005] Multiple aspects of the present application provide a head-mounted smart device, a streaming media processing method, a storage medium and a program product, so as to provide a new type of end-side AI device and enrich the form of the end-side AI device.

[0006] An embodiment of the present application provides a head-mounted smart device, comprising: a first headband portion, a second headband portion, two earmuff portions, a display screen and a circuit board, wherein the circuit board at least comprises a processing chip for running a target application on the head-mounted smart device and providing model reasoning capability for the target application; the circuit board is arranged on the first headband portion, and the first headband portion is connected to the two earmuff portions, so as to wear the head-mounted smart device on the user's head; the two earmuff portions are electrically connected to the processing chip, so as to pick up or output an audio signal related to the target application; the display screen is arranged on the second headband portion, and the second headband portion is connected to the two earmuff portions and can move toward the user's face to drive the display screen to be within the user's field of view; the display screen is electrically connected to the processing chip, so as to display screen data related to the target application.

[0007] An embodiment of the present application also provides a streaming media processing method, which is applied to the head-mounted smart device provided in the embodiment of the present application, and the method includes: running a target application on the head-mounted smart device; during the running of the target application, using a display screen to display screen data related to the target application, and / or using two earmuffs to pick up or output audio signals related to the target application.

[0008] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0009] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0010] In an embodiment of the present application, a head-mounted end-side AI device is provided, including two headbands and earmuffs. A circuit board portion including a processing chip is disposed on one headband portion, which is responsible for providing model reasoning capabilities. A display screen is disposed on another headband portion. The headband portion provided with the display screen has the ability to move toward the user's face, and can drive the display screen to move within the user's field of vision. The audio signal related to the target application is picked up or output through the earmuffs to solve the perception problem of AI. The head-mounted intelligent device of this embodiment meets the requirements of AI for perception, reasoning and display capabilities, and is different from the implementation form of traditional end-side AI devices. It is a new type of end-side AI device, which enriches the implementation form of end-side AI devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1a A schematic diagram of the structure of a head-mounted smart device provided as an exemplary embodiment of the present application;

[0013] Figure 1b This is a structural schematic diagram of a second headband portion provided in another exemplary embodiment of the present application being in a second state;

[0014] Figure 1c This is a structural schematic diagram of a second headband portion provided in another exemplary embodiment of the present application being in a third state;

[0015] Figure 2 A schematic diagram of the structure of a head-mounted smart device provided by another exemplary embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of a head-mounted smart device provided as another exemplary embodiment of the present application;

[0017] Figure 4a A schematic diagram of the structure of a head-mounted smart device provided as another exemplary embodiment of the present application;

[0018] Figure 4b A schematic diagram of the structure of a head-mounted smart device provided as another exemplary embodiment of the present application;

[0019] Figure 4c A schematic diagram of the structure of a head-mounted smart device provided as another exemplary embodiment of the present application;

[0020] Figure 4d A schematic diagram of the structure of a head-mounted smart device provided as another exemplary embodiment of the present application;

[0021] Figure 5 A structural diagram of a data processing engine provided for another exemplary embodiment of the present application;

[0022] Figure 6 A flowchart of a streaming media processing method provided for yet another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0025] In addition, it should be noted that, in the case where the embodiments of the present application involve user interaction operations or trigger operations, the user interaction operations or trigger operations involved in the embodiments of the present application include but are not limited to: touch operation, gesture operation, voice operation, head movement operation, eye movement operation and other interactive operations in various ways; among which, touch operation includes but is not limited to: click operation, double-click operation, long press operation, sliding operation, pinch operation or mouse hover operation, etc. Sliding operation includes but is not limited to: straight line sliding, curve sliding, etc.

[0026] Furthermore, it should be noted that, in the case where the embodiments of the present application involve jumping between the first interface and the second interface, the jumping methods involved in the embodiments of the present application include but are not limited to: jumping directly from the first interface to the second interface, jumping from the first interface to the task interface first and jumping to the second interface after completing the corresponding task operation on the task interface; completing the corresponding task operation on the task interface includes but is not limited to: when the task interface is implemented as a game interface, completing the game operation on the game interface; when the task interface is implemented as an identity authentication interface, completing the identity authentication on the identity authentication interface; when the task interface is implemented as a recharge interface, completing the recharge operation on the recharge interface; and so on.

[0027] With the evolution of AI technology, the form of application programs has also changed dramatically. Compared with traditional applications, AI applications have evolved. Among them, traditional applications can be regarded as rule-driven applications, referred to as rule applications. Rule applications rely on rules and logic programming, that is, the implementation of functions in rule applications depends on the rules predefined by developers. However, the essence of AI applications is intelligent applications driven by AI algorithms. Compared with rule applications, the intelligence of AI applications is mainly reflected in the improvement of perception and reasoning decision-making capabilities, so the corresponding modules of the end-side AI devices need to be improved accordingly.

[0028] Among them, in the embodiment of the present application, a perception module corresponding to the perception capability and a reasoning and decision-making module corresponding to the reasoning and decision-making capability are provided for the AI ​​device. The improvement of the perception module is reflected in that it relies on the improvement of the intelligence of the reasoning and decision-making module on the one hand; on the other hand, it relies on the acquisition of user data from multiple dimensions. The multiple dimensions include multi-modality and multi-time period. Multi-modality is reflected in the type of data, for example, it may include text, voice and video; multi-time period is reflected in the time range, for example, it may be night time, day time or all-weather time. In addition, the implementation of the reasoning and decision-making module is to accelerate the operation of the AI ​​algorithm, which puts forward various hardware resource requirements such as computing power, memory, storage, etc. for the end-side AI device; and the data obtained by the perception module is processed in real time based on the AI ​​algorithm, so the real-time requirements of the end-side AI device are relatively high.

[0029] In view of the above, in an embodiment of the present application, a head-mounted intelligent device is provided, including two headband parts and earmuff parts. The circuit board part including the processing chip is arranged on one headband part, which is responsible for providing model reasoning capabilities. The display screen is arranged on the other headband part. The headband part provided with the display screen has the ability to move toward the user's face, and can drive the display screen to move within the user's field of view; the audio signal related to the target application is picked up or output through the earmuff part to solve the perception problem of AI. The head-mounted intelligent device of this embodiment meets the requirements of AI for perception, reasoning and display capabilities, and is different from the implementation form of traditional end-side AI devices. It is a new type of end-side AI device, which enriches the implementation form of end-side AI devices.

[0030] Furthermore, in the embodiments of the present application, the head-mounted smart device also has the advantages of being small, light, and comfortable to wear.

[0031] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0032] Figure 1a The following is a schematic diagram of the structure of a head-mounted smart device provided by an exemplary embodiment of the present application. Figure 1aAs shown, the head-mounted smart device includes: a first headband portion 11, a second headband portion 12, two earmuff portions 13 (only one is shown in the figure), a display screen 121 and a circuit board 111. The circuit board 111 includes at least a processing chip for running a target application on the head-mounted smart device and providing model reasoning capabilities for the target application. The model reasoning capabilities can be regarded as reasoning and decision-making capabilities. Among them, the first headband portion 11 and the second headband portion 12 adopt a relatively independent design, that is, the first headband portion 11 and the second headband portion 12 are independent of each other and do not affect each other.

[0033] In this embodiment, the first headband portion 11 and the second headband portion are designed to be relatively independent, and the relative position between the first headband portion 11 and the second headband portion 12 is not limited. For example, it can be as follows Figure 1a As shown in the figure, the first headband portion 11 is located inside the second headband portion 12; or, it can also be outside the first headband portion 11. In each figure, the first headband portion 11 is located inside the second headband portion 12 as an example for explanation, but it does not mean that the first headband portion 11 can only be located inside the second headband portion 12.

[0034] In this embodiment, the first headband portion 11 is arranged inside the second headband portion 12. The first headband portion 11 can be regarded as the headband of the head-mounted smart device. When the user wears the head-mounted smart device, the first headband portion 11 can be located on the top of the user's head. The first headband portion 11 is connected to the two earmuff portions 13 and can serve as a supporting structure of the head-mounted smart device, used to support the head-mounted smart device to be worn on the user's head; and a circuit board 111 is also arranged on the first headband portion 11.

[0035] Furthermore, a display screen 121 is provided on the second headband portion 12, and is connected to two earmuffs 13 together with the first headband portion 11. The second headband portion 12 can move toward the user's face to drive the display screen on the second headband portion to move within the user's field of vision. Figure 1a The second headband portion is rotated to the user's field of view as an example, and Figure 1a The state of the second headband portion is called the first state, and the first state refers to the state of the second headband portion when it moves toward the user's face. In particular, since the first headband portion 11 and the second headband portion 12 are designed to be relatively independent, it is ensured that even when the second headband portion 12 moves toward the user's face, it will not be interfered by the first headband portion 11.

[0036] Figure 1b An example is given in which the second headband portion is in the second state. The second state can be regarded as the initial state of the second headband portion, that is, the state in which the second headband portion has not started to move when the user wears the headband-type smart device. Figure 1bThe second headband portion and the first headband portion overlap and maintain a certain distance in the vertical direction as an example of the initial state. In actual use, there are many possibilities for the initial state of the second headband portion. For example, after the user wears it, the display screen on the second headband portion is within the user's field of vision. At this time, the state of the second headband portion can also be called the initial state. Therefore Figure 1b The initial state shown is only an example and does not limit the present embodiment. Figure 1b As shown, although the first headband portion and the second headband portion overlap, since the two are relatively independent designs, the second headband portion is located outside the first headband portion. Further, the second headband portion 12 can be arranged along the Figure 1b The direction of the bidirectional curve, that is, Figure 1b The movement direction of the second headband part is toward the direction relative to the user's face, so as to drive the display screen on the second headband part to move into the user's field of view, and then Figure 1a As shown in FIG. 1 , it is an example of the state of the second headband portion during rotation, namely, the first state. It should be understood that the first state is any state of the second headband portion during activity. Further, when the second headband portion drives the display screen to move within the user's field of view, that is, when the display screen is in a better display position, the activity can be stopped. At this time, the state of the second headband portion is called the third state. Figure 1c The figure shows an example of the third state of the second headband portion.

[0037] The control method of the movement of the second headband is not limited. For example, the user can manually move the second headband; or the second headband on the head-mounted smart device can be automatically controlled based on the corresponding mechanical structure and driver chip.

[0038] Furthermore, in this embodiment, a target application is running on the head-mounted smart device, and the target application may be various application programs. Preferably, the target application includes various applications that rely on AI algorithms (such as machine learning tasks), such as map applications, note applications, language processing applications, e-commerce applications, health management applications, home control applications, and event reminder applications, etc. In the case where the target application is an application that relies on AI algorithms (such as machine learning tasks), the target application may also be referred to as an AI application. In the case where the AI ​​application relies on machine learning tasks, the head-mounted smart device is required to have model reasoning capabilities.

[0039] For example: Map applications can provide navigation assistance based on machine learning tasks, such as AR (Augmented Reality) navigation, which can provide path guidance and other relevant traffic information for users when walking or driving; note-taking applications can record meetings and extract meeting summaries based on machine learning tasks; event reminder applications can record events and notify events based on machine learning tasks; language processing applications can perform real-time translation based on machine learning tasks, and translate conversations between users and others in real time; health management applications can help users monitor their physical health based on machine learning tasks, and provide targeted health advice, such as exercise goal setting and diet control; shopping applications can overlay detailed information on products in the user's field of view to help users understand and experience the products; and home control applications can realize automated control of smart home devices based on machine learning tasks, such as controlling lights, air conditioning temperature, and TV switches.

[0040] Among them, the difference between the target application and the rule application includes its perception ability, reasoning and decision-making ability and display ability. With respect to the reasoning and decision-making ability, in this embodiment, a processing chip is configured for the head-mounted intelligent device, and the processing chip is arranged on a circuit board. There is no limitation on the type of the processing chip. Any processor that can run an AI algorithm can be used as the processing chip of this embodiment. For example, it can be a processor or computing module designed specifically for artificial intelligence computing. Such a processor or computing module can be referred to as an AI processing chip. The AI ​​processing chip can accelerate the execution of machine learning tasks and focus on improving the computing performance required to execute AI algorithms. For example, it can be an NPU chip (Neural Processing Unit) or a TPU chip (Tensor Processing Unit), but is not limited thereto. In this embodiment, the processing chip can run the target application, and can provide the target application with model reasoning ability when the target application relies on machine learning tasks. It can also respond to user or environmental changes in real time, that is, the need for real-time performance, and provide a hardware foundation for the head-mounted intelligent device to quickly complete calculations locally.

[0041] Further, with respect to perception capability, in this embodiment, the target application can realize auditory perception and visual perception with the user based on the processing chip. Wherein, when realizing auditory perception with the target application based on the processing chip, the processing chip is connected to two earmuffs, and the two earmuffs are used to pick up or output audio signals related to the target application; wherein, picking up refers to picking up the voice signal emitted by the user; or, the ambient sound of the user's environment can also be picked up; output refers to the audio content output by the head-mounted smart device to the user. Further, when realizing visual perception with the target application based on the processing chip, the processing chip can be electrically connected to the display screen, and the display screen is used to display the screen data related to the target application. Optionally, when realizing visual perception with the target application based on the processing chip, a camera can be added to the head-mounted smart device, which can be used to adopt the user's motion or environmental information. The camera can be implemented as a monocular camera, a binocular camera, etc., without limitation. Optionally, when the visual perception of the target application is realized based on the processing chip, a preset number of cameras can be set on each of the two earmuffs, for example, one camera can be set on each earmuff, and then the processing chip is connected to the two earmuffs to pick up the video signal related to the target application, wherein picking up refers to picking up the action initiated by the user; or, it can also pick up the information of the user's environment, etc.

[0042] In an embodiment of the present application, a head-mounted end-side AI device is provided, including two headbands and earmuffs. A circuit board portion including a processing chip is disposed on one headband portion, which is responsible for providing model reasoning capabilities. A display screen is disposed on another headband portion. The headband portion provided with the display screen has the ability to move toward the user's face, and can drive the display screen to move within the user's field of vision. The audio signal related to the target application is picked up or output through the earmuffs to solve the perception problem of AI. The head-mounted intelligent device of this embodiment meets the requirements of AI for perception, reasoning and display capabilities, and is different from the implementation form of traditional end-side AI devices. It is a new type of end-side AI device, which enriches the implementation form of end-side AI devices.

[0043] In an embodiment of the present application, two earmuffs of the head-mounted smart device serve as components commonly connected to the first headband portion and the second headband portion. An implementation structure of the earmuffs is given below.

[0044] In an alternative embodiment, if Figure 2As shown, the earmuff portion 13 includes an earmuff shell 131, which serves as an external structural component of the earmuff portion 13 and is usually made of plastic, metal or other durable materials. Its main function is to provide support and protection for the internal components of the earmuff portion 13, such as a speaker unit, a microphone and a vibration module. Among them, the speaker unit can be used to output audio signals related to the target application; the microphone can be used to pick up audio signals related to the target application; the vibration module can be used to provide tactile feedback to the user, and the intensity, frequency and pattern of the vibration can be flexibly adjusted.

[0045] In addition, the earmuff shell 131 is also used to install the first headband part and the second headband part. Among them, the earmuff shell 131 is provided with a mounting groove 132, and the mounting groove 132 can be implemented as a groove structure. The first headband part and the second headband part are installed in the mounting groove, and the first headband part is installed on the inner side of the second headband part, and when the second headband part is in the second state, the first headband part and the second headband part overlap and are spaced a certain distance apart, so that a relatively independent design between the first headband part and the second headband part can be achieved, reducing mutual interference between the two, and facilitating relative movement between the two. Optionally, the first headband part is fixedly installed in the mounting groove, and the second headband part is installed on a rotating mechanism on the head-mounted smart device, and the rotating mechanism is fixedly installed in the mounting groove. The rotating mechanism includes a rotating shaft and a connecting part connected to the rotating shaft, and the connecting part is fixedly connected to the end of the second headband. The first headband part and the second headband part are respectively introduced in detail below.

[0046] For the implementation structure of the first headband part, see Figure 3 The first headband includes a retractable first bracket 112. The two ends of the first bracket 112 are installed on the two earmuffs to provide support for the head-mounted smart device, so that the head-mounted smart device can fit closely to the user's head and remain stable. At the same time, the first headband also provides an up and down adjustment function. The first bracket in the first headband can be retracted up and down when worn. The direction of the up and down retraction can refer to Figure 3 The direction indicated by the double-direction straight arrow in the figure refers to the up and down telescoping relative to the user's head direction, that is, it allows telescopic adjustment along the axis perpendicular to the user's head (i.e. the up and down direction) to ensure that the head-mounted smart device can adapt to the height changes of different users' heads and improve the user's wearing comfort.

[0047] Among them, in this embodiment, the implementation structure of the first bracket 112 is not limited. In an optional embodiment, the first bracket is a slide rail structure, and the slide rail structure includes one or more rails, such as but not limited to a single slide rail or a double slide rail. In another optional embodiment, the first bracket can also be implemented as a non-slide rail structure, such as a curved plate, which is a curved plate-like structure. Compared with the slide rail structure, the curved plate can provide a larger contact surface, thereby more evenly dispersing the pressure of the head-mounted smart device and improving wearing comfort.

[0048] Further, in the present embodiment, the middle area of ​​the first bracket 112 includes an installation shell 113, wherein, with respect to the position of the installation shell on the first bracket, when the first bracket is implemented as a slide rail structure, the installation shell is located in the middle area of ​​the slide rail structure; similarly, when the first bracket is implemented as a non-slide rail structure, such as an arc-shaped plate, the installation shell is correspondingly located in the middle area of ​​the non-slide rail structure, such as the middle area of ​​the arc-shaped plate.

[0049] Optionally, a pressure buffer 115 is provided at the bottom of the mounting shell. The pressure buffer 115 is made of a flexible material to reduce pressure or impact during use and increase the durability and comfort of the head-mounted smart device. Specifically, on the one hand, when the head-mounted device is hit or falls, the pressure buffer can be used to reduce the impact of external force on the circuit board, thereby protecting the circuit board; on the other hand, when the user uses the head-mounted smart device, there is a situation where the head is fitted for a long time. The pressure buffer made of a flexible material can alleviate the discomfort caused by long-term contact to a certain extent and improve the user's wearing comfort. Optionally, the pressure buffer can be implemented as a load-bearing sponge or EPS (polystyrene) buffer; wherein the load-bearing sponge is a soft and elastic foam material that can deform to absorb energy when subjected to external force and return to its original state when the external force is removed. Because it is soft and elastic, it can be used in scenarios that require buffering, support and comfortable contact; EPS is a lightweight foam material made of polystyrene, which has excellent energy absorption capacity and is good at strong impact protection.

[0050] In this embodiment, the circuit board 111 can be arranged inside the mounting housing 113, thereby fixing and protecting the circuit board. Further, the first bracket 112 is provided with a first hollow cavity 114 on both sides of the mounting housing 113. The hollow cavity refers to a cavity with a hollow internal space, and the first hollow cavity 114 is used to carry a circuit electrically connected to the circuit board 111.

[0051] In this embodiment, the reasoning and decision-making capabilities of the target application are mainly provided by the processing chip on the circuit board 111. Therefore, the electrical connection with the circuit board 111 can be regarded as the interconnection with the processing chip on the circuit board 111. For example, the components interconnected with the processing chip may include but are not limited to the components in the earmuff part (such as speakers, microphones), display screens, cameras, etc.

[0052] Further optionally, the circuit board 111 may also include at least one of the following components: a memory chip, a power chip, a communication chip, a motion sensing sensor, and a physiological state sensing sensor. Among them, the sensing capabilities of the target application are mainly the responsibility of the motion sensor and the physiological state sensing sensor. The motion sensing sensor is used to detect the user's motion state and position changes, for example, it may include but not be limited to a gyroscope, a speedometer, an accelerometer, a GPS (Global Positioning System), etc.; the physiological state sensing sensor is used to detect the user's physiological parameters, for example, the physiological parameters may be but are not limited to body surface temperature, heart rate, blood pressure, etc.; wherein different physiological parameters can be collected using corresponding sensors. In the case where the circuit board further includes at least one of the above components, the interconnection with the circuit board 111 may be interconnection with at least one of the above components. It should be understood that in this embodiment, in order to meet the requirements of lightweight head-mounted smart devices, the circuit board can be designed to be more compact.

[0053] In an optional embodiment, the circuit board 111 also includes a display interface, a data interface and other external expansion interfaces; wherein the display interface is used to output the picture data on the display screen; the data interface is used to store and transmit data; other external expansion interfaces may include but are not limited to a communication interface, a camera interface and an audio interface, etc.; wherein the communication interface is used to expand the communication function of the head-mounted smart device; the camera interface is used to connect the camera; the audio interface is used for audio input and output.

[0054] For the implementation structure of the second headband part, such as Figure 4a As shown, optionally, the two ends of the second bracket are installed on the two earmuff parts, and the second headband part includes a retractable second bracket 122. The specific implementation of the retractability of the second bracket 122 is different from that of the first bracket. The retractability of the second bracket 122 is reflected in the front and back extension, that is, after the second headband part starts to rotate from the initial state (that is, the second state), the start of rotation can be regarded as the second headband part is in the first state, that is, the state in which the second headband part is in the process of moving toward the user's face. On this basis, the second bracket is retracted and retracted back and forth relative to the user's face. Specifically, when the head-mounted smart device is in the wearing state, the second bracket can be at the top of the user's head, that is, in the second state, and the first bracket and the second bracket overlap at this time, but are relatively independent, and the first bracket and the second bracket are spaced a certain distance apart. Then, the second bracket can be moved along as shown in the figure. Figure 4a The direction shown by the bidirectional curve in the middle, i.e., the forward and backward rotation direction of the second headband portion, rotates forward and backward around the user's head. During the forward and backward rotation, the second headband portion is in the first state. The forward and backward rotation causes the second bracket to drive the display screen thereon to rotate toward the user's field of view, and then the second bracket can be in the third state as described above after the forward and backward rotation.

[0055] During the forward and backward rotation of the second bracket, when the second bracket is rotated to the front of the user's field of vision, the second bracket can be moved along the following Figure 4a The direction indicated by the bidirectional straight line in the middle is the front-to-back extension direction of the second headband part, which is extended and retracted relative to the user's face. The distance between the display screen on the second bracket and the user's eyes can be adjusted by retracting and retracting, that is, the focal length can be adjusted, thereby improving the visual clarity and ensuring that different users have a good visual experience during use. In this embodiment, Figure 4b As shown, after rotating and extending forward and backward, the second bracket is within the user's field of view, that is, in the third state. The third state can be regarded as the final state of the second bracket and is also the best position for the user to watch the picture on the display screen.

[0056] It should be noted that, in this embodiment, after the second bracket is extended and retracted forward and backward, it can also be rotated forward and backward to adjust the display screen to a better display position, that is, the second bracket is in the third state.

[0057] Among them, in this embodiment, the way to achieve telescopic movement of the first bracket and the second bracket is not limited. The following is an introduction to the specific implementation method of the front and rear telescopic movement of the second bracket, and the first bracket can also be telescopically moved in the same way. In an optional embodiment, the second headband portion includes a main body and an end portion; a groove is provided at the end portion, and a limiting portion for limiting the main body is provided on the side wall of the groove, and the main body abuts against the end groove and can slide in the groove; the limiting portion can abut against the bottom of the main body at the position where the sliding stops as the main body slides, so as to fix the main body. Among them, the implementation structure of the limiting portion is not limited, and it can be a series of retractable protrusions. When the main body slides and stops at the protruding position, the protrusion pops up to abut against the bottom of the main body, and when it leaves the protruding position, it shrinks into the corresponding protruding groove; it can also be a slide rail linked to the main body, and a contact portion is provided at the end of the slide rail. When the main body stops sliding, the contact portion pops up to abut against the bottom of the main body to fix the main body.

[0058] In this embodiment, the display capability of the target application is mainly provided by the display screen on the second bracket. An example of the display screen being provided at the second bracket is given below. Figure 4bAs shown, in an optional embodiment, the display screen is arranged in the middle area of ​​the second bracket, and the second bracket on both sides of the display screen is provided with a second hollow cavity 123 for carrying a circuit electrically connected to the display screen.

[0059] Among them, in this embodiment, the display screen is an auxiliary design, and the purpose of setting the display screen is to display information, so as to assist users to obtain corresponding information from a visual perspective in real time. For example, the screen data related to the target application can be displayed on the display screen, and the screen data can be presented on the display screen in the form of text, pictures, videos, etc. Among them, for different types of target applications, the display screen can display corresponding pictures. For example, for map applications, real-time route prompts can be displayed. During the real-time route prompt process, the display screen can display key information such as the direction, distance, and speed of the user's upcoming turn, so as to help users understand the current road conditions in real time; for language processing applications, such as real-time translation applications, on the one hand, the head-mounted smart device can use the microphone in the earmuff to obtain the content of the conversation between the user and others, and translate the content of the conversation and display it on the display screen, so as to help the user understand the content of the conversation.

[0060] Of course, the display screen can also display some information about the head-mounted smart device itself, such as but not limited to battery power, peripheral connection status (such as the connection status of a Bluetooth headset), volume and playback status, etc., without limitation.

[0061] Optionally, the display screen is an edge curved screen. An edge display screen is a display screen with curved edges, which can extend the display area of ​​the display screen to the edge of the display screen and can bend backward. The edge display screen increases the field of view and the sense of envelopment of the display screen, thereby achieving more efficient information transmission.

[0062] In an optional embodiment, in order to reduce the weight of the head-mounted smart device, the thickness of the display screen is limited to 0.8 mm-1.5 mm. Preferably, in the design process, an optimal balance is achieved between the display effect and the weight of the head-mounted smart device, and the thickness of the display screen can be 1.0 mm.

[0063] Further optionally, a cover 124 is provided on the outside of the display screen, and the cover 124 is a protective and functional component; the protective function is reflected in that the cover 124 can cover the outside of the display screen, play a role in waterproofing, dustproofing and physical impact protection, and extend the service life of the display screen; the functionality is reflected in that it can be made of specific optical materials, which can reduce light reflection and glare during use, and ensure that users can see the images presented on the display screen clearly under different lighting conditions. Among them, the cover 124 can be fixedly connected to or detachably connected to the second bracket, and there is no limitation on this.

[0064] Further optionally, the head-mounted smart device also includes a camera for providing the target application with the ability to collect video signals. Figure 4c As shown, the camera 133 can be set on the earmuff part 13. The camera set on the earmuff part can be consistent with the user's line of sight. When the user moves his line of sight, the camera can follow in time to maintain consistency with the user's field of vision. Optionally, the camera can be set on the earmuff shell 131 of the earmuff part 13. For example, the head-mounted smart device can be implemented as a binocular camera. In this case, a camera can be set in the middle area of ​​each earmuff shell 131 in the vertical direction. The setting position of the camera on the earmuff shell 131 in any earmuff part 13 can refer to Figure 4c The binocular camera can be used to collect binocular video signals for the target application. The binocular camera can provide more powerful environmental perception capabilities and can greatly improve the capabilities of head-mounted smart devices in depth estimation, scene recognition, object recognition, and character recognition.

[0065] In an optional embodiment, the head-mounted smart device further includes: a touch sensor 134, which is disposed on the outside of the earmuff portion or on the edge area of ​​the second headband portion close to the earmuff portion. Figure 4d As shown, it is an example that the touch sensor 134 is arranged on the outside of the earmuff part, but it is not limited to this.

[0066] Among them, the touch sensor is used to sense the user's touch operation, and convert the touch operation into a touch instruction and provide it to the processing chip, so that the processing chip can respond to the touch instruction. In this embodiment, the type of touch sensor is not limited, for example, including but not limited to capacitive, resistive, infrared and ultrasonic types. The touch sensor works by detecting physical contact on its surface. When the touch sensor surface senses the user's touch operation, the physical change can be converted into a corresponding touch instruction and provided to the processing chip. When the processing chip receives the touch instruction, it will perform a corresponding operation according to the touch instruction. The operation can be related to the target application, such as quickly starting or shutting down the target application; or, it can also be some operations for the head-mounted smart device, such as adjusting the output volume of the earmuff, adjusting the brightness and resolution of the display screen, etc. The touch operation method can be but is not limited to: single tap, double tap or multiple taps on the device location where the touch sensor is located (such as the earmuff); it can also be a single long press, or intermittent multiple long presses on the device location where the touch sensor is located (such as the earmuff); different touch operations can correspond to different touch instructions, for example, a single tap corresponds to opening the target application, a double tap corresponds to closing the target application, a long press corresponds to increasing the output volume of the earmuff, and so on.

[0068] In the above embodiments, the hardware structure of the head-mounted smart device provided in the embodiments of the present application is introduced. The head-mounted smart device provided in the embodiments of the present application is compared with the traditional XR head-mounted display device. Among them, XR (Extended Reality, XR) is a general term for AR and VR. In the process of comparing with the traditional XR (Extended Reality) head-mounted display device that mainly focuses on multimedia experience, the traditional XR head-mounted display device does not support end-side AI on the one hand, and at least requires end-cloud collaboration, or even realizes cloud-based AI; on the other hand, these head-mounted display devices are committed to providing users with high-quality picture display effects; and, they also need to provide better sound quality, and have higher requirements for the accuracy of perception sensors, etc. These demands to enhance the user's multimedia experience make traditional head-mounted display devices large in size, heavy in weight, and cause discomfort when worn for a long time.

[0069] However, in an embodiment of the present application, the head-mounted smart device runs the target application. When the target application is implemented as an intelligent application, such as an application that relies on machine learning tasks, it is an application that supports data flow tasks (i.e., an application that supports streaming processing). This application does not need to store data streams in real time, but analyzes and processes data streams in real time, and does not require high picture rendering capabilities. Therefore, when selecting various components or chips for head-mounted smart devices, there are certain advantages, such as selecting a storage chip with a smaller storage capacity, and selecting an NPU chip dedicated to the target application. Under the same power consumption, the computing performance of the NPU chip running the AI ​​algorithm is more powerful than that of the traditional CPU chip, and a low-power display screen can also be selected, so that the head-mounted smart device has the characteristics of lightweight. In addition, based on the advantage of lightweight head-mounted smart devices, head-mounted smart devices can be designed as a structure similar to headphones that can be carried and worn comfortably, and can adapt to different usage environments. Furthermore, on the basis of the lightweight component selection, the hardware energy consumption of the head-mounted smart device is lower, so that a longer battery life can be provided. For example, in actual tests, it was found that the weight of the head-mounted smart device can be controlled within 300g (grams), the peak power consumption can be within 5W (watts), the average power consumption is within 2W, and the standby power consumption is within 0.2W. Furthermore, in order to increase the endurance of the head-mounted smart device, a larger capacity battery can be selected for the head-mounted smart device, for example, the battery capacity range is: 1000mah (milliampere hour) ~ 2000mah (milliampere hour), the voltage range is 3.5V ~ 4.5V, but not limited to this.

[0070] In an embodiment of the present application, a set of data processing engines adapted to the head-mounted smart device in the above embodiment is also provided. The data processing engine is introduced below in conjunction with the accompanying drawings.

[0071] like Figure 5 As shown, the embodiment of the present application provides a data processing engine, which is deployed on one side of the head-mounted smart device 50, so it can be called a data processing engine 51 on the terminal side. Among them, the data processing engine 51 is located between the operating system 53 of the head-mounted smart device 50 and the target application 52, and provides the required runtime environment for the target application 52 with the help of the operating system 53. Specifically, it means using the various software and hardware resources of the head-mounted smart device 50 with the help of the operating system 53 of the head-mounted smart device 50, such as the processing chip 541, the storage chip 542, the communication chip 543 and the audio chip 544, and providing the target application 52 with the runtime environment required for its operation based on these software and hardware resources. The runtime environment involves some basic capabilities required for the processing chip to run the target application, such as image processing capability, audio and video processing capability, model reasoning capability, etc.

[0072] In this embodiment, there is no restriction on the number and type of processing chips included in the head-mounted smart device 50. For example, the head-mounted smart device 50 may include only one processing chip, and the type of the processing chip may be, but is not limited to, an NPU chip, a TPU chip, and a GPU chip, etc.; for another example, the head-mounted smart device 50 may include multiple processing chips, and the types of the multiple processing chips may be the same or different, and the type of any processing chip may be, but is not limited to, an NPU chip, a TPU chip, and a GPU chip, etc.

[0073] Figure 5 The head-mounted smart device includes multiple processing chips 541 as an example, which does not limit this embodiment. In this embodiment, the processing chip 541 mainly refers to an AI processing chip. At the same time, the head-mounted smart device can also include other computing resources, such as a CPU chip. In addition, in this embodiment, multiple head-mounted smart devices 50 are also provided, and the processing chips used by each head-mounted smart device 50 can be different, so the data processing engine supports multiple types of processing chips.

[0074] In this embodiment, the target application 52 supports at least one data stream task. The data stream task refers to a task that uses machine learning technology to process data streams. The data stream can be at least one of an audio stream, a video stream, and a text stream. Among them, the data processing object of the data stream task can be end-side user behavior data, such as but not limited to voice signals picked up by the earmuffs; or, it can also be video data captured by a camera on a head-mounted smart device; or, it can also be corresponding sensor data captured by other types of sensors, which are not listed here one by one. Optionally, the operating system 53 provides a sensor acquisition interface 531, through which the target application 52 can access various sensor data.

[0075] In an embodiment of the present application, according to the processing operation on the data stream, the data stream task is divided into multiple subtasks, and one subtask represents a processing operation on the data stream. Afterwards, the multiple subtasks are interpreted and executed in sequence, and finally the processing operation on the data stream can be completed. In this embodiment, the data stream task includes multiple subtasks that act serially on the same data object in the data stream. When the data stream is implemented as a video stream, the data object is any video frame; when the data stream is implemented as a text stream, the data object can be text data, such as a word, a sentence, or a paragraph, etc.; when the data stream is implemented as an audio stream, the data object is any audio frame. Accordingly, the subtask is a machine learning task. When the data stream task is a video stream task, the subtask can be, but is not limited to, an image processing subtask, a model reasoning subtask, and an image rendering subtask, etc.

[0076] It should be noted that in the subsequent embodiments, the data flow task will be a video flow task, the data object is a video frame, and the data flow task includes an image processing subtask, a model reasoning subtask and an image rendering subtask to illustrate the process of the data processing engine executing the data flow task, but it is not limited to this. In this embodiment, during the operation of the target application 52, the data processing engine 51 can be responsible for executing the data flow task, and the execution of the data flow task involves a large number of tensor calculation tasks. Among them, the data processing engine 51 includes: a runtime scheduler 511, a dynamic programming language interpreter 512, a machine learning library 513 and a tensor calculation engine 514; the main work of the tensor calculation engine 514 is: tensor definition, tensor calculation implementation and optimization. The execution process of the tensor calculation engine 514 is generally: (1) using operators to construct a calculation graph; (2) tensor calculation graph; (3) given an input tensor, the tensor calculation engine 514 executes the calculation graph to obtain and return the output tensor. In this embodiment, a tensor is a vector or matrix of any dimension, and tensor calculation is a calculation process with a tensor as input and output. The tensor computing engine 514 is mainly used to provide basic tensor computing capabilities to the outside world using tensor operators (Operators, Ops) as units. The tensor computing engine 514 can be, but is not limited to, an MNN (Mobile Neural Network) engine, which is a lightweight deep learning end-side reasoning engine that includes optimization, conversion, and reasoning of deep neural network models, and has the characteristics of high versatility and high performance.

[0077] Furthermore, the machine learning library 513 in the data processing engine 51 is responsible for providing some library functions required for machine learning tasks, and data flow tasks can be written based on the library functions. That is to say, data flow tasks can use the library functions in the machine learning library 513. Among them, the machine learning library 513 is constructed based on the tensor operator provided by the tensor computing engine 514, and an integrated design is implemented for the machine learning libraries that different subtasks rely on, which is conducive to promoting the unification of data types, data structures and underlying operators, and realizing zero data copy between library functions (or modules) of different machine learning libraries. Among them, each subtask can be decomposed into one or more tensor operators, etc. In addition, the machine learning library 513 can be one or more, and there is no limitation on this. Figure 5 The machine learning libraries are used as examples, which do not constitute a limitation to the present embodiment.

[0078] In this embodiment, multiple machine learning libraries correspond to multiple subtasks in the data flow task. For example, when the multiple subtasks are image processing subtasks, model reasoning subtasks, and image rendering subtasks, the machine learning libraries corresponding to the multiple subtasks are image processing library, model reasoning library, and image rendering library. In other words, the machine learning library constructed based on the tensor computing engine includes at least one of the following: image processing library, model reasoning library, and image rendering library.

[0079] Among them, in this embodiment, there is no limitation on the dynamic programming interpretation language for writing data flow tasks, and it can include, for example, Python (a widely used interpreted, high-level and general programming language), JavaScript (a lightweight, interpreted or just-in-time compiled programming language with function-first characteristics) and Ruby (a simple and fast object-oriented programming scripting language), etc.

[0080] Furthermore, during the operation of the target application 52, the runtime scheduler 511 can interpret and execute the triggered data flow tasks assigned to the corresponding processing chip of the head-mounted smart device 50 according to the characteristics of each subtask. In an optional embodiment, the head-mounted smart device includes a CPU chip, and the CPU chip participates in the corresponding calculation and processing. Optionally, the head-mounted smart device 50 includes an AI processing chip and a CPU chip, and the AI ​​processing chip and the CPU chip can be integrated together; alternatively, the AI ​​processing chip can also exist as a separate chip, which is not limited.

[0081] Further optionally, in the case where the head-mounted smart device 50 includes multiple processing chips, the runtime scheduler can allocate multiple subtasks in the triggered target data stream task to multiple target processing chips of the head-mounted smart device 50 according to the characteristics of each subtask during the operation of the target application. Different subtasks will be allocated to appropriate target processing chips to avoid conflicts between target processing chips, and the types of target processing chips of different subtasks can be the same or different, depending on the characteristics of the subtasks and the types of processing chips supported by the head-mounted smart device 50. It is explained here that the above-mentioned multiple target processing chips can be part or all of the hardware computing resources included in the head-mounted smart device.

[0082] In an optional embodiment, the types of the multiple target processing chips are the same. In this case, since there is no difference in the types of the target processing chips assigned to the subtasks, during the assignment process, the operating performance, occupancy rate, and throughput of the multiple processing chips on the head-mounted smart device can be combined to assign the multiple subtasks to the appropriate target processing chips, thereby improving resource utilization.

[0083] In another optional embodiment, the types of multiple target processing chips may be different. In the process of assigning target processing chips to subtasks, the matching degree between subtasks and processing chips may be considered. That is to say, any subtask has its own matching degree with different types of processing chips. In the process of assigning a target processing chip to any subtask, any subtask is assigned to a processing chip of a type with a higher matching degree, and the processing chip is used as the target processing chip. For example, when the subtask is a model reasoning subtask, it can be assigned to an NPU chip for interpretation and execution; when the subtask is an image rendering subtask, it can be assigned to a GPU chip for interpretation and execution. Among them, assigning different subtasks to different processing chips is conducive to reducing the burden of a single processing chip, providing a basis for parallel scheduling of multiple subtasks, and improving the execution efficiency of subtasks in data flow tasks while improving the utilization of hardware computing resources of head-mounted intelligent devices.

[0084] In this embodiment, the runtime scheduler is also used to obtain the data stream corresponding to the target data stream task, and schedule multiple subtasks acting on different video frames in the data stream in a pipeline parallel manner. Optionally, the data associated with the data stream task can be collected through the camera of the terminal device and various other sensors. The data corresponding to the data stream includes multiple data objects, and the data object can include at least one of audio frames, video frames and text data.

[0085] In this embodiment, the data object included in the subtask is a video frame for example. For multiple subtasks, the video frame can be used as the granularity to act on each video frame or part of the video frame, and the part of the video frame can be a sampled or selected video frame. Among them, pipeline parallelism refers to scheduling and executing multiple subtasks acting on the same video frame in a pipeline manner, and multiple subtasks acting on different video frames can be executed in parallel to improve the overall execution efficiency of the subtask. For example, if the data flow task is a video flow task, its processing process will be relatively repetitive, including multiple video frames, and the processing process of each video frame can be divided into several sequentially executed subtasks, such as image processing, model reasoning and image rendering. Each subtask can be assigned an independent processing chip, and the subtasks on different video frames will be assigned to different processing chips. For each processing chip, the subtask it is responsible for can be executed independently, and the multiple processing chips responsible for multiple subtasks on the same video frame can be sequentially operated, and the multiple processing chips responsible for subtasks on different video frames can be operated in parallel, so that the efficient processing of data flow tasks can be achieved in a resource-constrained environment.

[0086] Wherein, the subtask is a section of program code written in a dynamic programming language. The dynamic programming language interpreter 512 in the data processing engine 51 is adapted thereto. The dynamic programming language interpreter 512 refers to an interpreter that can interpret and execute program codes programmed in a dynamic programming language, and can also be called a virtual machine for a dynamic programming language. Wherein, the dynamic programming language used to write the subtask is different, and the dynamic programming language interpreter will also be different. The dynamic programming language interpreter includes but is not limited to the Python interpreter, the JavaScript interpreter, and the Ruby interpreter.

[0087] In this embodiment, the dynamic language interpreter 512 in the data processing engine 51 is used to interpret and execute the scheduled subtasks based on the machine learning library 513 and the tensor computing engine 514 according to the scheduling order of multiple subtasks acting on different video frames by the runtime scheduler.

[0088] In the embodiment of the present application, a machine learning library corresponding to multiple subtasks in a data flow task is constructed based on a tensor computing engine, so that multiple subtasks do not need to interact between multiple engines during the interpretation and execution process, thereby improving the running performance of the application. In addition, the runtime scheduler is responsible for allocating multiple subtasks to appropriate target processing chips and scheduling multiple subtasks in a pipeline parallel manner to improve computing efficiency and resource utilization.

[0089] Optionally, for any data flow task that is scheduled, the data flow task is interpreted as a machine instruction, and the machine instruction points to the target library function in the machine learning library 513 corresponding to the data flow task. Further, the tensor computing engine 514 is called to execute the target tensor operator corresponding to the target library function on the processing chip to which the data flow task is assigned to execute the data flow task. The tensor computing engine 514 executes the target tensor operator to generate tensor data as an output result. After the tensor operators corresponding to all library functions corresponding to the data flow task are executed, the generated tensor data is stored in the memory space of the target application as the execution result data of the data flow task.

[0090] Figure 6 A flow chart of a streaming media processing method provided in an embodiment of the present application. The method is applied to a head-mounted smart device in the above embodiment of the claim, and the method comprises:

[0091] S601: running a target application on the head-mounted smart device;

[0092] S602: During the running of the target application, the screen data related to the target application is displayed using the display screen, and / or the audio signals related to the target application are picked up or output using the two earmuffs.

[0093] In an optional embodiment, running the target application on the head-mounted smart device includes: using the two earmuffs to receive a voice signal emitted by a user; if the voice signal is recognized to indicate starting the target application, running the target application; and during the running of the application, using a data processing engine to interpret and execute a data flow task triggered in the target application.

[0094] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0095] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of step 601 to step 602 can be device A; for another example, the execution subject of step 601 can be device A, and the execution subject of step 602 can be device B; and so on.

[0096] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 601, 602, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0097] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0098] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0099] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0100] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0101] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0102] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0103] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0105] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0106] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0108] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0109] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0110] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0112] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A head-mounted smart device, characterized in that: include: A first headband portion, a second headband portion, two earmuff portions, a display screen and a circuit board, wherein the circuit board includes at least a processing chip for running a target application on the head-mounted smart device and providing a model reasoning capability for the target application; The circuit board is arranged on the first headband part, and the first headband part is connected to the two earmuff parts, so as to wear the head-mounted smart device on the head of the user; the two earmuff parts are electrically connected to the processing chip, so as to pick up or output the audio signal related to the target application; The display screen is arranged on the second headband part, the second headband part is connected to the two earmuff parts and can be moved toward the user's face to drive the display screen to be within the user's field of vision; the display screen is electrically connected to the processing chip for displaying screen data related to the target application.

2. The device according to claim 1, characterized in that The processing chip is also used to: run a data processing engine, which is located between the operating system of the head-mounted smart device and the target application, and is used to provide a runtime environment required by the target application.

3. The device according to claim 2, characterized in that The processing chip is specifically used for: receiving voice signals emitted by the user through the two earmuffs, and running the target application if the voice signal is recognized to indicate starting the target application; and during the running of the target application, using the data processing engine to interpret and execute the data flow tasks triggered in the target application.

4. The device according to claim 1, characterized in that The first headband portion includes a retractable first bracket, two ends of the first bracket are mounted on the two earmuff portions, and the first bracket can be retracted up and down in a wearing state; The middle area of ​​the first bracket includes a mounting shell, the circuit board is located inside the mounting shell, and the first bracket is provided with a first hollow cavity on both sides of the mounting shell for carrying a circuit electrically connected to the circuit board.

5. The device according to claim 4, characterized in that The first bracket is a slide rail structure, and the mounting shell is arranged in a middle area of ​​the slide rail structure.

6. The device according to claim 4, characterized in that A pressure buffer portion is provided at the bottom of the mounting shell, and the pressure buffer portion is made of a flexible material.

7. The device according to claim 6, characterized in that The pressure buffer part is implemented as a load-bearing sponge or an EPS buffer part.

8. The device according to claim 1, characterized in that The second headband portion includes a retractable second bracket, both ends of which are mounted on the two earmuffs, and the second bracket can rotate forward and backward in a wearing state and can be retracted forward and backward relative to the user's face; The display screen is installed in the middle area of ​​the second bracket, and the second bracket is provided with a second hollow cavity at the two sides of the display screen for carrying a circuit electrically connected to the display screen.

9. The device according to claim 8, characterized in that A cover body is arranged on the outer side of the display screen, and the cover body is fixedly connected or detachably connected to the second bracket.

10. The device according to claim 8, characterized in that The display screen is an edge curved screen.

11. The device according to any one of claims 1 to 10, characterized in that: The thickness of the display screen is in the range of 0.8 mm to 1.5 mm.

12. The device according to any one of claims 1 to 10, characterized in that The earmuff part comprises an earmuff shell, a mounting groove is provided on the earmuff shell, the first headband part and the second headband part are mounted in the mounting groove, and the first headband part is mounted on the inner side of the second headband part.

13. The device according to any one of claims 1 to 10, characterized in that The circuit board also includes at least one of the following components: a storage chip, a power chip, a communication chip, a motion sensing sensor, and a physiological state sensing sensor.

14. The device according to any one of claims 1 to 10, characterized in that The device also includes: a touch sensor, which is arranged on the outside of the earmuff part or on the edge area of ​​the second headband part close to the earmuff part, for sensing the user's touch operation, and converting the touch operation into a touch instruction and providing it to the processing chip, so that the processing chip responds to the touch instruction.

15. A streaming media processing method, characterized in that: Applied to the head-mounted smart device according to any one of claims 1 to 13, the method comprising: Running the target application on the head-mounted smart device; During the operation of the target application, the screen data related to the target application is displayed by the display screen, and / or the audio signals related to the target application are picked up or output by the two earmuffs.

16. The method according to claim 15, characterized in that Running the target application on the head-mounted smart device includes: Using the two earmuffs to receive voice signals sent by the user; If the voice signal is recognized as indicating starting the target application, running the target application; During the running of the application program, the data processing engine is used to interpret and execute the data flow tasks triggered in the target application program.

17. A readable storage medium, characterized in that: include: A computer program / instructions which, when executed by a processor, causes the processor to implement the steps of claim 15 or 16.

18. A program product, characterized in that include: A computer program / instructions which, when executed by a processor, causes the processor to implement the steps of claim 15 or 16.

Images