Collaborative work system, method and electronic device

Abstract

Embodiments of the present application provide a kind of collaborative work system, method and electronic equipment.In the collaborative work system, including electronic equipment and internet of things equipment, the camera for being used to gather image downward is arranged on the internet of things equipment, the target application in the electronic equipment is bound with the internet of things equipment, the virtual camera corresponding to the camera of internet of things equipment is registered in the system by the electronic equipment, then it can be realized by calling the virtual camera to call the camera of internet of things equipment and gather image.Relative to the photograph of user handheld electronic equipment, the picture of internet of things equipment is more stable, and it will not affect the related operation when the target application in the electronic equipment is used by user.This way, electronic equipment and internet of things equipment collaborative work, can bring better use experience to user, and internet of things equipment does not need to have display function, also reduce the cost of internet of things equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent terminals, in particular to a collaborative work system and method and electronic device. BACKGROUND

[0002] With the development of the Internet, online education is becoming more and more popular. Online education is not limited by time and place, and is more visual and attractive in terms of pictures and audio. How to provide better online education experience for users based on intelligent devices is a problem to be solved. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application provide a collaborative work system, method and electronic device. In the collaborative work system, the electronic device and the Internet of Things device work collaboratively, which can bring better user experience, and the Internet of Things device does not need to have display function, which also reduces the cost of the Internet of Things device.

[0004] In a first aspect, the embodiments of the present application provide a collaborative work system, comprising: an electronic device and an Internet of Things device, the Internet of Things device being provided with a camera, the camera being used to collect image data downward, and a target application in the electronic device being bound with the Internet of Things device;

[0005] The electronic device is used to register a virtual camera corresponding to the camera in the system, and send an image preview request to the Internet of Things device by calling the virtual camera;

[0006] The Internet of Things device is used to collect preview image data by calling the camera according to the image preview request of the electronic device, and send the preview image data to the target application in the electronic device for preview display;

[0007] The electronic device is further used to generate a photographing request when calling the virtual camera, and send the photographing request to the Internet of Things device;

[0008] The Internet of Things device is further used to photograph an image by calling the camera according to the photographing request of the electronic device, and send the photographed image data to the target application in the electronic device for display.

[0009] The target application can be any application program that needs to call the camera, such as the education application mentioned below.

[0010] In this way, the electronic device and the Internet of Things device work collaboratively, which can bring better user experience. Compared with the user holding the electronic device to take pictures, the Internet of Things device collects pictures more stably, and does not affect the related operation of the user using the target application in the electronic device. Moreover, the Internet of Things device does not need to have display function, which also reduces the cost of the Internet of Things device.

[0011] According to a first aspect, the electronic device is further configured to: before registering the virtual camera corresponding to the camera in the system, verifying the IoT device, connecting the IoT device, and obtaining whether the IoT device has the capability of supporting the virtualized camera;

[0012] The electronic device is further configured to: when it is determined that the IoT device has the capability of supporting the virtualized camera, register the virtual camera corresponding to the camera in the system.

[0013] According to the first aspect, or any one of the implementations of the first aspect, the electronic device comprises a camera service, a virtual camera HAL, a hardware virtualization service, and a transmission management service; the IoT device comprises a device interconnection service; the transmission management service and the device interconnection service are configured to provide the capability of interconnecting the electronic device and the IoT device;

[0014] The electronic device is configured to: transmit the image preview request to the device interconnection service of the IoT device through the camera service, the virtual camera HAL, the hardware virtualization service, and the transmission management service in sequence, and transmit the received preview image data to the target application through the transmission management service, the hardware virtualization service, the virtual camera HAL, and the camera service in sequence for display; wherein the target application is adapted to the hardware virtualization service and the transmission management service.

[0015] In this way, the image preview function of the target application is realized based on the camera service of the device system, and the program modification amount of the target application can be reduced.

[0016] According to the first aspect, or any one of the implementations of the first aspect, the electronic device is configured to: transmit the photographing request to the device interconnection service of the IoT device through the hardware virtualization service and the transmission management service in sequence, and transmit the received photographing image data to the target application through the transmission management service and the hardware virtualization service in sequence for display.

[0017] In this way, the photographing function of the target application is realized based on the private protocol, the transmission path of the photographing request and the photographing image is shorter, and the time delay is reduced.

[0018] According to the first aspect, or any one of the implementations of the first aspect, the IoT device is a table lamp.

[0019] In this way, the electronic device and the table lamp work cooperatively, and a better online education experience can be provided for the user.

[0020] According to the first aspect, or any one of the implementations of the first aspect, the electronic device is configured to: in response to a first operation, register the virtual camera corresponding to the camera in the system, the first operation being configured to start the word lookup function in the target application; and the target application is authenticated by the electronic device.

[0021] The electronic device is further configured to perform image recognition on the preview image, determine position information of the finger or the pointing tool in the preview image, and determine a ROI image in the preview image according to the position information; and the pointing tool is used to point to a word or a picture in a book.

[0022] The electronic device is further configured to perform image recognition on the ROI image, determine the word to be explained or the picture to be explained, and display an explanation corresponding to the word to be explained or the picture to be explained.

[0023] For example, the first operation can be an operation of clicking a query function option in the target application.

[0024] For example, the pointing tool can be a point reading pen, various pens (pencils, carbon pens, etc.), or other similar cylindrical tools.

[0025] In this way, the electronic device and the desk lamp work together to provide a user with a better word query function use experience.

[0026] According to the first aspect or any one of the implementations of the first aspect, the electronic device is configured to, in response to a second operation, register a virtual camera corresponding to the camera in the system, the second operation being used to start a submission function in the target application; and the target application is authenticated by the electronic device.

[0027] The electronic device is further configured to, in response to a third operation, generate a photographing request when the virtual camera is called, send the photographing request to the Internet of Things device, and display a photographed image returned by the Internet of Things device.

[0028] The electronic device is further configured to, in response to a fourth operation, upload the photographed image to a database.

[0029] For example, the second operation can be an operation of clicking a homework function option in the target application, the third operation can be an operation of clicking a photographing option, and the fourth operation can be an operation of clicking a submission option.

[0030] In this way, the electronic device and the desk lamp work together to provide a user with a better homework submission function use experience.

[0031] According to the first aspect or any one of the implementations of the first aspect, the electronic device is configured to, in response to a fifth operation, register a virtual camera corresponding to the camera in the system, the fifth operation being used to start a pointing function in the target application; and the target application is authenticated by the electronic device.

[0032] The electronic device is further configured to perform image recognition on the preview image, determine a book name, and load book content according to the book name.

[0033] The electronic device is further configured to: perform image recognition on the preview image, determine position information of the finger or the pointing tool in the preview image and a book page number, determine first content in the book content according to the book page number and the position information, and read the first content; or perform image recognition on the preview image, determine a book page number, determine second content in the book content according to the book page number, and read the second content.

[0034] For example, the fifth operation can be an operation of clicking a reading function option in the target application.

[0035] For example, the pointing tool can be a reading pen, various pens (pencils, carbon pens, etc.), or other similar cylindrical tools.

[0036] The first content and the second content can both be understood as a paragraph to be read.

[0037] According to the first aspect or any one of the implementations of the first aspect, the electronic device includes a mobile phone or a tablet computer.

[0038] In a second aspect, an embodiment of the present application provides a method for cooperative work, including:

[0039] The electronic device registers a virtual camera corresponding to the camera in the Internet of Things device in the system, and sends an image preview request to the Internet of Things device by calling the virtual camera; and a target application in the electronic device is bound to the Internet of Things device.

[0040] The Internet of Things device collects preview image data by calling the camera according to the image preview request of the electronic device, and sends the preview image data to the target application in the electronic device for preview display.

[0041] In response to a user operation, the electronic device generates a photographing request when calling the virtual camera, and sends the photographing request to the Internet of Things device.

[0042] The Internet of Things device photographs an image by calling the camera according to the photographing request of the electronic device, and sends the photographed image data to the target application in the electronic device for display.

[0043] According to the second aspect, before the electronic device registers the virtual camera corresponding to the camera in the system, the method further includes: the electronic device verifies the Internet of Things device, connects the Internet of Things device, and obtains whether the Internet of Things device has the capability of supporting the virtualized camera.

[0044] The electronic device registers the virtual camera corresponding to the camera in the system can include: the electronic device registers the virtual camera corresponding to the camera in the system when it is determined that the Internet of Things device has the capability of supporting the virtualized camera.

[0045] According to a second aspect, or any possible implementation mode of the second aspect, the electronic device comprises a camera service, a virtual camera HAL, a hardware virtualization service, a transmission management service; the Internet of Things device comprises a device interconnection service; the transmission management service and the device interconnection service are used to provide the capability of interconnection between the electronic device and the Internet of Things device.

[0046] The electronic device sends an image preview request to the Internet of Things device, which can comprise that the electronic device transmits the image preview request to the device interconnection service of the Internet of Things device through the camera service, the virtual camera HAL, the hardware virtualization service and the transmission management service in sequence.

[0047] The electronic device receives the preview image data returned by the Internet of Things device and displays in the target application, which can comprise that the received preview image data is transmitted to the target application for display through the transmission management service, the hardware virtualization service, the virtual camera HAL and the camera service in sequence.

[0048] The electronic device sends a photographing request to the Internet of Things device, which can comprise that the electronic device transmits the photographing request to the device interconnection service of the Internet of Things device through the hardware virtualization service and the transmission management service in sequence.

[0049] The electronic device receives the photographing image data returned by the Internet of Things device and displays in the target application, which can comprise that the received photographing image data is transmitted to the target application for display through the transmission management service and the hardware virtualization service in sequence.

[0050] According to the second aspect, or any possible implementation mode of the second aspect, the Internet of Things device is a table lamp.

[0051] According to the second aspect, or any possible implementation mode of the second aspect, the method further comprises:

[0052] The electronic device registers a virtual camera corresponding to the camera in the system in response to a first operation, the first operation being used to start the word searching function in the target application; the target application is authenticated by the electronic device.

[0053] The electronic device performs image recognition on the preview image, determines the position information of the finger or the word pointing tool in the preview image, and determines the ROI image in the preview image according to the position information; the word pointing tool is used to point to the text or the picture in the book.

[0054] The electronic device performs image recognition on the ROI image, determines the text to be explained or the picture to be explained, and displays the explanation corresponding to the text to be explained or the picture to be explained.

[0055] According to the second aspect, or any possible implementation mode of the second aspect, the method further comprises:

[0056] The electronic device registers, in response to a second operation, a virtual camera corresponding to the camera in the system, the second operation being used to start a submission job function in the target application; the target application is authenticated by the electronic device;

[0057] The electronic device generates, in response to a third operation, a photographing request when invoking the virtual camera, sends the photographing request to the Internet of Things device, and displays a photographed image returned by the Internet of Things device;

[0058] The electronic device uploads, in response to a fourth operation, the photographed image to a database.

[0059] According to a second aspect, or any possible implementation mode of the second aspect, the method further includes:

[0060] The electronic device registers, in response to a fifth operation, a virtual camera corresponding to the camera in the system, the fifth operation being used to start a point reading function in the target application; the target application is authenticated by the electronic device;

[0061] The electronic device performs image recognition on the preview image, determines a book name, and loads book content according to the book name;

[0062] The electronic device performs image recognition on the preview image, determines position information of a finger or a finger tool in the preview image and a book page number, determines first content in the book content according to the book page number and the position information, and reads the first content; or the electronic device performs image recognition on the preview image, determines a book page number, determines second content in the book content according to the book page number, and reads the second content.

[0063] According to the second aspect, or any possible implementation mode of the second aspect, the electronic device includes a mobile phone or a tablet computer.

[0064] The second aspect and any possible implementation mode of the second aspect correspond to the first aspect and any possible implementation mode of the first aspect respectively. The technical effects of the second aspect and any possible implementation mode of the second aspect can refer to the technical effects of the first aspect and any possible implementation mode of the first aspect, which will not be described here.

[0065] In a third aspect, an electronic device is provided. The electronic device includes one or more processors, memory, and one or more computer programs stored in the memory, which when executed by the one or more processors, cause the electronic device to perform the cooperative method of the method steps performed by the electronic device in the second aspect and any possible implementation mode of the second aspect.

[0066] The third aspect and any kind of implementation manner of the third aspect correspond to the second aspect and any kind of implementation manner of the second aspect respectively. The technical effects corresponding to the third aspect and any kind of implementation manner of the third aspect can be referred to the technical effects corresponding to the second aspect and any kind of implementation manner of the second aspect, which will not be described here.

[0067] In a fourth aspect, the embodiments of the present application provide an Internet of Things device. The Internet of Things device comprises one or more processors; a memory; a camera; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the Internet of Things device performs the cooperative working method composed of the method steps executed by the Internet of Things device in the second aspect and any kind of implementation manner of the second aspect.

[0068] The fourth aspect and any kind of implementation manner of the fourth aspect correspond to the second aspect and any kind of implementation manner of the second aspect respectively. The technical effects corresponding to the fourth aspect and any kind of implementation manner of the fourth aspect can be referred to the technical effects corresponding to the second aspect and any kind of implementation manner of the second aspect, which will not be described here.

[0069] In a fifth aspect, the embodiments of the present application provide a computer readable storage medium. The computer readable storage medium comprises a computer program, when the computer program is run on an electronic device, the electronic device performs the cooperative working method composed of the method steps executed by the electronic device in the second aspect and any kind of implementation manner of the second aspect, or the Internet of Things device performs the cooperative working method composed of the method steps executed by the Internet of Things device in the second aspect and any kind of implementation manner of the second aspect.

[0070] The fifth aspect and any kind of implementation manner of the fifth aspect correspond to the second aspect and any kind of implementation manner of the second aspect respectively. The technical effects corresponding to the fifth aspect and any kind of implementation manner of the fifth aspect can be referred to the technical effects corresponding to the second aspect and any kind of implementation manner of the second aspect, which will not be described here.

[0071] In a sixth aspect, the embodiments of the present application provide a computer program product, comprising a computer program, when the computer program is run, the computer performs the cooperative working method composed of the method steps executed by the electronic device in the second aspect and any kind of implementation manner of the second aspect, or the cooperative working method composed of the method steps executed by the Internet of Things device in the second aspect and any kind of implementation manner of the second aspect.

[0072] The sixth aspect and any one of the implementation manners of the sixth aspect correspond to the second aspect and any one of the implementation manners of the second aspect respectively. For details of the technical effects of the sixth aspect and any one of the implementation manners of the sixth aspect, refer to the technical effects of the second aspect and any one of the implementation manners of the second aspect, which will not be described herein again.

[0073] In a seventh aspect, the present application provides a chip, comprising a processing circuit, a transceiving pin. Wherein, the transceiving pin and the processing circuit communicate with each other through an internal connection path, the processing circuit executes the cooperative working method composed of the method steps executed by the electronic device in the second aspect and any one of the implementation manners of the second aspect, or executes the cooperative working method composed of the method steps executed by the Internet of Things device in the second aspect and any one of the implementation manners of the second aspect, to control the receiving pin to receive signals, to control the sending pin to send signals.

[0074] The seventh aspect and any one of the implementation manners of the seventh aspect correspond to the second aspect and any one of the implementation manners of the second aspect respectively. For details of the technical effects of the seventh aspect and any one of the implementation manners of the seventh aspect, refer to the technical effects of the second aspect and any one of the implementation manners of the second aspect, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figures 1a-1b An application scenario is exemplarily shown;

[0076] Figure 2a A hardware structure schematic diagram of an electronic device is exemplarily shown;

[0077] Figure 2b A software structure schematic diagram of an electronic device is exemplarily shown;

[0078] Figure 3a A hardware structure schematic diagram of an Internet of Things terminal is exemplarily shown;

[0079] Figure 3b A software structure schematic diagram of an Internet of Things terminal is exemplarily shown;

[0080] Figure 4a A module interaction schematic diagram provided by the embodiment of the present application is exemplarily shown;

[0081] Figure 4b A module interaction schematic diagram provided by the embodiment of the present application is exemplarily shown;

[0082] Figures 5a-5b An application scenario is exemplarily shown;

[0083] Figure 6a A module interaction schematic diagram provided by the embodiment of the present application is exemplarily shown;

[0084] Figure 6b A module interaction schematic diagram provided for an embodiment of the present application;

[0085] Figures 7a-7b An application scenario shown for example;

[0086] Figure 8 A module interaction schematic diagram provided for an embodiment of the present application;

[0087] Figures 9a-9c An application scenario shown for example;

[0088] Figure 10 A module interaction schematic diagram provided for an embodiment of the present application

[0089] Figures 11a-11b An application scenario shown for example. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0091] The term “and / or” in the present document is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone.

[0092] The terms “first” and “second” and the like in the description and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0093] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of the words “exemplary” or “for example” is intended to present concepts in a specific manner.

[0094] In the description of the embodiments of the present application, unless otherwise specified, the meaning of “a plurality of” is two or more. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0095] With the development of the internet, online education has become increasingly popular, and users (such as students) have a growing demand for it. In some applications, students can look up unfamiliar words online for explanations; in others, online reading aloud of textbook content facilitates learning and pronunciation; and still others require students to submit their assignments online. Therefore, how to meet users' online education needs using smart devices is a problem that needs to be solved.

[0096] Currently, users in online education scenarios typically use smart learning devices that combine camera and display functions. These devices require a camera or reflector in a specific position to photograph the book, limiting their versatility and ease of use. Furthermore, devices with both camera and display functions require robust hardware and system support, resulting in high costs. Therefore, how to provide users with a better online education experience based on smart devices, improve versatility and ease of use, and reduce the cost of online education are problems that need to be addressed.

[0097] This application provides a collaborative work system applicable to online education scenarios. The system includes an electronic device and a desk lamp with a communication connection. The desk lamp is equipped with a camera that can be used to photograph books. The electronic device uses the desk lamp's camera to capture images and combines them with online education resources on the platform to meet the user's online education needs. The electronic device can be a tablet or mobile phone, etc. Besides online education scenarios, the electronic device and desk lamp can also serve users based on their respective basic functions (i.e., communication and lighting). In this way, the system can create a better online education experience for users based on two commonly used smart devices, offering strong versatility and ease of use. Furthermore, since tablets or mobile phones are already essential products in most households, and the cost of a desk lamp with only a camera function is lower than that of a device with both camera and display functions, it significantly reduces the cost of online education for users.

[0098] The following explanation uses a tablet computer as an example to illustrate the technical solution provided in this application.

[0099] Figure 1a An example application scenario is shown. For example... Figure 1aAs shown, the cooperative working system includes the tablet 100 and the desk lamp 200 which are connected. The desk lamp 200 includes a camera 201 for capturing images downward, for example, capturing the content of a book. The tablet 100 is installed with an education APP (application) which can call the camera 201 of the desk lamp 200 to capture images and provide various online education functions for the user according to the images captured by the camera 201 of the desk lamp 200, such as online word checking, online reading, online homework submission, etc.

[0100] Although the tablet also has a front camera and a rear camera, the user needs to hold the tablet to aim the camera at the book when capturing the book with either camera, which is not stable and affects the user's finger reading and point reading operation, and cannot provide a better online education experience for the user. Figure 1a As shown, the tablet 100 and the desk lamp 200 can be placed in fixed positions, and the tablet 100 captures the book with the camera 201 of the desk lamp 200, which is stable and has a high content recognition success rate, and the user can flexibly perform finger reading and point reading operation in the book. Thus, the cooperation of the tablet and the desk lamp can provide a better online education experience for the user.

[0101] As shown, Figure 1b The tablet 100 and the desk lamp 200 can communicate through near field communication or far field communication. The near field communication can complete information exchange between devices through a router or other device, and the far field communication can complete information exchange between devices through a cloud server. For example, the tablet 100 and the desk lamp 200 can realize near field communication based on a Wi-Fi (wireless fidelity) network protocol.

[0102] As shown, Figure 2a The structure of the electronic device 100 is shown in the figure. Optionally, the electronic device 100 can be a terminal, also known as a terminal device. The terminal can be a cellular phone or a pad, etc. The present application does not make any limitation. It should be noted that the structure of the electronic device 100 can be applied to the tablet in Figures 1a-1b It should be understood that the electronic device 100 shown in the figure is only an example of the electronic device, and the electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. Figure 2a The electronic device 100 shown in the figure is only an example of the electronic device, and the electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. Figure 2a The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0103] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, a barometric pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0104] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0105] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0106] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0107] The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0108] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 while also providing power to the electronic device through the power management module 141.

[0109] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160, etc.

[0110] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0111] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0112] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0113] The wireless communication module 160 can provide solutions for wireless communication, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and the like, which are applied to the electronic device 100.

[0114] In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology.

[0115] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor, and the like. The GPU is a microprocessor for image processing, connected with the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0116] The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 can include 1 or N display screens 194, N being a positive integer greater than 1.

[0117] In the embodiments of the present application, the display screen 194 can display a shooting preview interface and a photographing image interface, and the like. It should be noted that, in the embodiments of the present application, the shooting preview interface refers to an interface through which a user can view images collected by the desk lamp camera in real time.

[0118] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0119] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, for example, to enable the electronic device 100 to implement the cooperative work method in the embodiments of the present application. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created during use of the electronic device 100 (such as audio data, a phone book, etc.), and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0120] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, and the like. For example, music playing, recording, and the like.

[0121] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or part of the functions of the audio module 170 can be disposed in the processor 110.

[0122] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A. In some embodiments, the electronic device 100 can be provided with multiple speakers 170A.

[0123] The receiver 170B, also known as a "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the receiver 170B can be held close to the ear to listen to the voice.

[0124] Microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak into the microphone 170C by putting his mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, it can also realize the function of noise reduction. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, to realize the functions of collecting sound signals, noise reduction, identifying sound sources, realizing directional recording, etc.

[0125] The earphone interface 170D is used to connect the wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0126] The pressure sensor is used to sense the pressure signal, and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor.

[0127] The touch sensor, also called "touch panel". The touch sensor can be provided on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also called "touch screen". The touch sensor is used to detect the touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type.

[0128] The key 190 includes the power-on key (or power key), volume key, etc. The key 190 can be a mechanical key or a touch key. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0129] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and also can be used for touch vibration feedback. For example, the touch operation acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects.

[0130] The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of power, and also can be used to indicate messages, missed calls, notifications, etc.

[0131] The software system of the electronic device 100 can employ a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. Embodiments of the present application take an Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0132] Figure 2b is a software structure block diagram of the electronic device 100 of embodiments of the present application.

[0133] The layered architecture of the electronic device 100 divides the software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the system layer, the HAL layer (Hardware Abstract Layer), and the kernel layer.

[0134] The application layer can include a series of application packages.

[0135] As shown in Figure 2b , the application packages can include calls, videos, Bluetooth, cameras, WLAN, educational applications, device manager applications, etc. The application packages can also include calendar, map, navigation, music, short message, etc. applications.

[0136] Among them, the educational application can be used to provide online education functions for users, such as online word recognition, online reading, online homework submission, etc.

[0137] In some examples, the device manager application can be used to bind IOT (Internet of Things) devices such as desk lamps. In some examples, the educational application can implement the binding of IOT (Internet of Things) devices such as desk lamps.

[0138] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0139] As shown in Figure 2b , the application framework layer can include camera services, authentication services, hardware virtualization services, device management services, and transmission management services.

[0140] Among them, the camera service can be used to invoke the camera (including the front camera and / or the rear camera) in response to the request of the application.

[0141] In the embodiments of the present application, the camera service can be used to invoke the virtual camera on the electronic device side, i.e., the camera in the IOT device, in response to the request of the application.

[0142] The authentication service is used to provide security permission management capability.

[0143] The hardware virtualization service can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IOT device side, and provide the capability of virtualizing the camera.

[0144] The device management service can be used for IOT device discovery and management, and provide far-field (i.e., cloud) IOT device information and near-field (i.e., nearby connectable) IOT device information for applications such as education applications.

[0145] The transmission management service can be used to establish a physical transmission channel and provide data transmission capability.

[0146] In addition, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. can also be included.

[0147] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc.

[0148] The content provider is used to store and obtain data, and make the data accessible to applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying pictures, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including connection, hang-up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction.

[0149] The system library and the runtime layer (i.e., the system layer) include system libraries and Android runtime.

[0150] The Android runtime includes core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0151] The core library includes two parts: one part is the function function required to be called by the java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to execute the functions of object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0152] In the embodiment of the application, the Android Runtime also includes a virtual camera adaptation layer, which provides the ability of virtual camera registration.

[0153] The system library in the system layer can include a plurality of functional modules. For example: a multimedia platform, a graphics image processing library, a codec, etc.

[0154] The multimedia platform can be used for the management of multimedia, supports a plurality of commonly used audio, video format playback and recording, and static image files, etc. The multimedia platform can support a plurality of audio and video coding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0155] The graphics image processing library can be used to realize graphics drawing, image rendering, synthesis, and layer processing, etc.

[0156] The codec can be used to realize the coding and decoding operation of audio data and video data.

[0157] The HAL layer is an interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: an audio HAL, a sensor HAL, a modem HAL, a camera HAL, and a virtual camera HAL.

[0158] The audio HAL is used to process the audio stream, for example, to perform noise reduction, directional enhancement, etc. on the audio stream. The camera HAL is used to process the image stream corresponding to the side camera of the electronic device, and the virtual camera HAL is used to process the image stream corresponding to the virtual camera registered by the electronic device, that is, to process the image stream collected by the side camera of the IOT device.

[0159] The kernel layer is the layer between hardware and software. The kernel layer at least contains display driver, camera region, audio driver, network driver (such as Wi-Fi driver), CPU driver, USB driver, storage driver, and print driver, etc. The hardware at least includes processor, display screen, Wi-Fi module, etc.

[0160] It can be understood that, Figure 2bThe layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer layers than illustrated, and each layer can include more or fewer components, which are not limited in the present application.

[0161] As Figure 3a Fig. 6 shows a hardware structure diagram of the Internet of Things device 200. It should be noted that the structure diagram of the Internet of Things device 200 can be applicable to the table lamp in Figures 1a-1b It should be understood that Figure 3a The Internet of Things device 200 shown is only an example of an electronic device, and the Internet of Things device 200 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. Figure 3a The various components shown in Figs. 1-6 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0162] The Internet of Things device 200 can include a processor 210, a camera 201, a wireless communication module 202, a memory 203, an audio module 204, a USB interface 205, a charging management module 206, a power management module 207, a battery 208, an illuminating device 209, a key 211, and the like.

[0163] The processor 210 can include one or more processing units, for example: the processor 210 can include a GPU, an ISP, a controller, a memory, a video codec, and the like. Different processing units can be independent devices or integrated into one or more processors.

[0164] The controller can be the nerve center and command center of the Internet of Things device 200. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions.

[0165] The camera 201 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive elements can be charge coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) phototransistors. The photosensitive elements convert the optical signals into electrical signals, which are then transmitted to the ISP to convert into digital image signals. The ISP outputs the digital image signals to the DSP for processing. Taking the table lamp as an example, the camera 201 can be arranged on the table lamp support to collect images downward.

[0166] The Internet of Things device 200 can implement a shooting function through an ISP, a camera 201, a video codec, a GPU, etc.

[0167] The ISP is used to process data fed back by the camera 201. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and conversion into a visible image. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 201.

[0168] The wireless communication module 202 can provide a wireless communication solution including WLAN (such as a Wi-Fi network), Bluetooth (BT), etc. applied to the Internet of Things device 200. In some embodiments, the antenna of the Internet of Things device 200 is coupled with the wireless communication module 202, so that the Internet of Things device 200 can communicate with the network and other devices through wireless communication technology.

[0169] The memory 203 can be used to store computer executable program codes including instructions. The processor 210 executes various functions of the Internet of Things device 200 and data processing by running the instructions stored in the memory 203, for example, to enable the Internet of Things device 200 to implement the cooperative working method in the embodiments of the present application.

[0170] The Internet of Things device 200 can implement an audio function such as music playing through an audio module 204, a loudspeaker 212, etc.

[0171] The USB interface 205 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 205 can be used to connect a charger to charge the Internet of Things device 200, and can also be used to transmit data between the Internet of Things device 200 and a peripheral device.

[0172] The charging management module 206 is used to receive charging input from the charger. The charging management module 206 charges the battery 208 while also providing power to the Internet of Things device 200 through the power management module 207.

[0173] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives input from the battery 208 and / or the charging management module 206 to provide power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, and the lighting device 209, etc.

[0174] The key 211 includes a power-on key (or power key) and the like.

[0175] The software system of the Internet of Things device 200 can adopt a layered architecture or other architecture, etc. The embodiment of the present application takes the layered architecture as an example to exemplarily illustrate the software structure of the Internet of Things device 200.

[0176] Figure 3b is the software structure block diagram of the Internet of Things device 200 of the embodiment of the present application.

[0177] The layered architecture of the Internet of Things device 200 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system of the Internet of Things device 200 is divided into three layers, from top to bottom, respectively, the application program layer, the application program framework layer, the system layer, and the kernel layer.

[0178] As shown in Figure 3b , the application program layer can include device application services, which can be understood as system-level applications, and the device application services start after the system of the Internet of Things device 200 starts.

[0179] As shown in Figure 3b , the application program framework layer can include device interconnection services, hardware abstraction services, resource managers, etc.

[0180] The device interconnection service can be used to establish a physical transmission channel, provide data transmission capability, and at the same time manage the start and stop of the hardware abstraction service.

[0181] The hardware abstraction service can be used to establish a logical channel between the electronic device side (i.e. the center device side) and the IOT device, provide the capability of virtual camera, and at the same time provide the camera open interface of the IOT device.

[0182] The resource manager can provide various resources for the application program.

[0183] As shown in Figure 3b , the system layer can include multimedia platforms, graphics image processing libraries, codecs, device adaptation modules, etc.

[0184] The multimedia platform can be used for the management of multimedia, supporting a variety of commonly used audio, video, and still image files, etc. The multimedia platform can support a variety of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0185] The graphics image processing library can be used to realize graphics drawing, image rendering, synthesis, and layer processing, etc.

[0186] The codec can be used to implement the codec operation on audio data, video data.

[0187] The device adaptation module can implement the interface of the hardware abstraction service, can provide device information and capability query, and can also provide functions of performing related operations on the IOT device side, such as functions of opening a camera, taking a photo, previewing, and the like.

[0188] It can be understood that, in order to implement the cooperative working method in the embodiments of the present application, the electronic device 100 and the IOT device 200 contain corresponding hardware and / or software modules for performing various functions. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or in the form of computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0189] As shown in Figure 4a is a schematic diagram of the interaction of each module. Referring to Figure 4a , the embodiments of the present application provide a method flow of cooperative working of a tablet and a desk lamp, specifically including:

[0190] 0. Device service initialization phase

[0191] S0.1, in response to a user operation, the device application service of the desk lamp is started, and the device interconnection service is loaded.

[0192] For example, the user operation can be an operation of the user turning on the power of the desk lamp. In response to the user operation, the desk lamp system is started, the device application service is started, and the device interconnection service is loaded. The device interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp, and can be used to provide data transmission capability.

[0193] S0.2, the device interconnection service of the desk lamp loads the hardware abstraction service.

[0194] The device interconnection service can also control the start of the hardware abstraction service. For example, after the device interconnection service is started, the device interconnection service can load the hardware abstraction service in the form of a plug-in. The hardware abstraction service can be used to establish a logical channel between the tablet and the desk lamp, to provide the capability of a virtual camera, and can also provide an open interface of the camera of the desk lamp.

[0195] Referring to Figure 4bThe illustrated module interaction diagram, the hardware abstraction service can include at least a base component and a camera component. Wherein, in the device service initialization phase, the device interconnection service loads the base component first, and initializes the base component. After the base component is initialized, information interaction can be carried out with the device adaptation module of the desk lamp to obtain device information and virtualization capability information. Exemplarily, the device information includes but is not limited to device name, device identification, device type, etc. Exemplarily, the virtualization capability information includes but is not limited to whether to support virtualization camera, whether to support virtualization microphone, etc.

[0196] Wherein, the desk lamp has the capability of supporting virtualization camera, which can be understood as that the camera of the desk lamp allows other electronic devices (such as tablet) to call, that is, the camera of the desk lamp can be understood as allowing to be used as a virtual camera of other electronic devices.

[0197] After the base component obtains the device information and capability information of the desk lamp, if the desk lamp has the capability of supporting virtualization camera, the base component loads the camera component to provide the capability of virtualization camera. At this time, the base component can proceed with the negotiation channel establishment preparation to wait for establishing the negotiation channel with the tablet to negotiate the network connection related information (including but not limited to IP address and port, etc.). Wherein, when the base component proceeds with the negotiation channel establishment preparation, a session service (Session Server) is created, and the session name of the session service is sent to the device interconnection service, so as to establish the negotiation channel between the transmission management service of the tablet side and the device interconnection service of the desk lamp.

[0198] 1. Device discovery phase

[0199] S1.1, in response to user operation, the education APP of the tablet sends a device discovery instruction to the device management service.

[0200] Wherein, the user operation can be an operation of the user clicking a function option in the education APP that needs to call a virtual camera. Exemplarily, the user operation can be an operation of the user clicking a reading function, a word checking function, a homework function, a photographing function in the education APP.

[0201] The education APP of the tablet receives the user operation, and in response to the operation, sends a device discovery instruction to the device management service of the tablet. Wherein, the device discovery instruction is used to indicate to find an IOT device capable of establishing a connection with the tablet. Exemplarily, the device discovery instruction can include but is not limited to instruction type and device type to be discovered. In this embodiment, the device discovery instruction is specifically used to find a desk lamp capable of establishing a connection with the tablet.

[0202] S1.2, the device management service in the tablet calls the authentication service to authenticate the education APP, and obtains an authentication result of the education APP.

[0203] After receiving the device discovery instruction, the device management service can obtain the name (or identifier) of the education APP based on the existing mechanism of the Android system, and perform APP authentication on the education APP according to the name of the education APP. The device management service can call the authentication service to authenticate the education APP to obtain an authentication result of the education APP.

[0204] In the embodiment, the device management API corresponding to the device management service and the hardware virtualization API corresponding to the hardware virtualization service are also provided in the tablet-side application framework layer. In order to implement the technical solution provided in the embodiment, the education APP needs to be registered on a related platform (for example, a platform provided by a tablet manufacturer), adapt to the framework of the device management service, the hardware virtualization service and the transmission management service, and apply for the permission of the device management API and the hardware virtualization API.

[0205] For example, the authentication service accesses the authentication server to authenticate the education APP through the authentication server, including but not limited to whether the education APP is registered on the related platform, whether the related framework is adapted, and whether the related API permission is applied.

[0206] For another example, the authentication service can authenticate the education APP according to a local white list.

[0207] After obtaining the authentication result (authentication success or authentication failure) of the education APP, the authentication service sends the authentication result to the device management service.

[0208] S1.3, the device management service in the tablet sends a device search instruction to the transmission management service when the education APP is authenticated successfully.

[0209] If the education APP is authenticated successfully, the device management service sends a device search instruction to the transmission management service. The device search instruction can include but is not limited to the instruction type, the device type to be searched and the search method. For example, the search method includes but is not limited to near-field device scanning and obtaining device information from a cloud server. In the embodiment, the device type to be searched is a desk lamp.

[0210] S1.4, the transmission management service in the tablet obtains a near-field device list according to the device search instruction, and sends the near-field device list to the device management service.

[0211] The far and near field device list includes a far field device list and a near field device list. The far field device included in the far field device list refers to a registered device obtained from the cloud server, and the near field device included in the near field device list refers to a device scanned through near field communication. In the far field device list and the near field device list, the device information includes but is not limited to device name, device identifier, device type, etc.

[0212] When the transmission management service receives the device search instruction, the device search operation is performed according to the device type to be searched and the search mode carried in the device search instruction, such as performing the near field device scanning operation and obtaining the related device information in the cloud server. The far field device list and the near field device list are obtained, and the far field device list and the near field device list are sent to the device management service.

[0213] S1.5, the device management service in the tablet performs device filtering according to the far and near field device list, and reports the device information obtained after filtering to the education APP.

[0214] The device management service performs device filtering according to the far field device list and the near field device list, determines the desk lamp information capable of being linked with the tablet, and sends it to the education APP. The device management service can perform the intersection operation on the far field device list and the near field device list, filter out the desk lamps existing only in the far field device list or only in the near field device list, and take the desk lamps existing in both the far field device list and the near field device list as the desk lamp information capable of being linked with the tablet. In this way, the device management service can filter out the desk lamps not registered in the cloud server, and also filter out the desk lamps unable to perform near field communication with the tablet.

[0215] In another optional embodiment, the transmission management service of the tablet can obtain the communication device list and the registered device list according to the device search instruction, regardless of whether the tablet and the desk lamp are in the same local area network. The devices included in the communication device list refer to the devices scanned through near field communication or far field communication, and the devices included in the registered device list refer to the registered devices obtained from the cloud server. The near field device included in the near field device list refers to the device scanned through near field communication. In the communication device list and the registered device list, the device information includes but is not limited to device name, device identifier, device type, etc.

[0216] The tablet transmits management service sends the communication device list and the registered device list to the device management service, the device management service performs device filtering according to the communication device list and the registered device list, and reports the device information obtained after filtering to the education APP. The device management service can perform an intersection operation on the communication device list and the registered device list, filter out the desk lamps that exist only in the communication device list or the registered device list, and take the desk lamps that exist in both the communication device list and the registered device list as the desk lamp information that can be linked with the tablet. In this way, the device management service can filter out the desk lamps that are not registered in the cloud server, and also filter out the desk lamps that cannot perform near field communication with the tablet.

[0217] 2. Virtual camera enabling phase

[0218] S2.1, the education APP of the tablet determines the desk lamp to be linked.

[0219] The number of desk lamps that can be linked with the tablet obtained by the device management service can be one or more. When the number of desk lamps is one, the education APP takes it as the desk lamp to be linked by default; when the number of desk lamps is more than one, the education APP can display a list of desk lamps to be linked for the user to select from, and in response to the selection operation of the user, the education APP takes the desk lamp selected by the user as the desk lamp to be linked.

[0220] It should be noted that the step of determining the desk lamp to be linked by the education APP can also be divided into the device discovery phase, which is not limited in the embodiment.

[0221] S2.2, the education APP of the tablet performs device verification and device connection on the desk lamp, and obtains the capability of the desk lamp to support virtual camera.

[0222] S2.3, the education APP of the tablet sends a virtual camera enabling request to the hardware virtualization service.

[0223] After the education APP of the tablet obtains the capability of the desk lamp to support virtual camera, the education APP sends a virtual camera enabling request to the hardware virtualization service. The virtual camera enabling request is used to indicate the registration of a virtual camera in the virtual camera HAL. The virtual camera enabling request can include but is not limited to request type, device name, device identifier, device type, and identifier of the virtual camera.

[0224] S2.4, the hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

[0225] After the hardware virtualization service receives the virtual camera enabling request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enabling request.

[0226] S2.5, after the virtual camera HAL of the tablet completes the virtual camera registration, sends a virtual camera enablement success indication to the education APP.

[0227] The flow of the virtual camera enablement phase will be explained in detail below in combination with the module interaction diagram shown in Figure 4b Referring to FIG. 3, the flow of the virtual camera enablement phase mainly includes a device verification sub-phase (S301-S309), a device connection sub-phase (S310-S321), a device service capability request sub-phase (S322-S325), and a virtual camera enablement sub-phase (S326-S331). Figure 4b

[0228] Referring to FIG. 3, the flow of the virtual camera enablement phase mainly includes a device verification sub-phase (S301-S309), a device connection sub-phase (S310-S321), a device service capability request sub-phase (S322-S325), and a virtual camera enablement sub-phase (S326-S331). Figure 4b

[0229] S301, the education APP in the tablet sends a virtual camera enablement instruction to the hardware virtualization API.

[0230] The virtual camera enablement instruction is used to indicate to enable the virtual camera. The virtual camera enablement instruction can include, but is not limited to, an instruction type, a device name, a device identifier, and a device type.

[0231] S302, after receiving the virtual camera enablement instruction, the hardware virtualization API in the tablet sends a device verification instruction to the interface scheduling module of the hardware virtualization service.

[0232] The device verification instruction is used to indicate to verify the device information carried in the virtual camera enablement instruction. The device verification instruction can include, but is not limited to, an instruction type, a device name, a device identifier, and a device type.

[0233] S303, the interface scheduling module of the hardware virtualization service sends an APP authentication instruction to the permission management module of the hardware virtualization service.

[0234] After receiving the device verification instruction, the interface scheduling module of the hardware virtualization service first sends an APP authentication instruction to the permission management module of the hardware virtualization service to authenticate the APP that initiates the virtual camera enablement instruction. The APP authentication instruction can include, but is not limited to, the name of the APP.

[0235] S304, the permission management module of the hardware virtualization service authenticates the education APP.

[0236] ​​The permission management module can access the authentication server to authenticate the education APP via the authentication server, including but not limited to authenticating whether the education APP is registered on a related platform, whether the education APP is adapted to a related framework, and whether the education APP applies for a related API permission. The permission management module can access the authentication server via the authentication service, which is not limited in the embodiment.

[0237] S305, when the permission management module of the hardware virtualization service in the tablet determines that the education APP is authenticated successfully, the permission management module sends an authentication success indication to the interface scheduling module.

[0238] After the permission management module obtains the authentication result of the education APP, if the education APP is authenticated successfully, the permission management module sends an authentication success indication to the interface scheduling module, and if the education APP is authenticated unsuccessfully, the permission management module sends an authentication failure indication to the hardware virtualization API, so that the hardware virtualization API returns APP has no permission indication information to the education APP according to the authentication failure indication.

[0239] S306, when the interface scheduling module of the hardware virtualization service in the tablet determines that the education APP is authenticated successfully, the interface scheduling module sends a device check instruction to the device management module.

[0240] After the interface scheduling module of the hardware virtualization service receives the authentication success indication and determines that the education APP is authenticated successfully, the interface scheduling module sends a device check instruction to the device management module. The device check instruction is used to check the state of the device to be linked, and in the embodiment, the device check instruction is used to check the state of the desk lamp to be linked. The device check instruction can include but is not limited to instruction type, device name, device identifier, and device type.

[0241] S307, the device management module of the hardware virtualization service in the tablet sends a device information query instruction to the device profile module of the device management service.

[0242] The device profile module of the device management service stores the device information currently online.

[0243] After the device management module of the hardware virtualization service receives the device check instruction, the device management module sends a device information query instruction to the device profile module of the device management service. The device information query instruction can include but is not limited to device name, device identifier, and device type.

[0244] S308, the device profile module of the device management service in the tablet returns device information to the device management module of the hardware virtualization service.

[0245] The device introduction module of the device management service returns the device information to the device management module of the hardware virtualization service according to the device information query instruction if the corresponding device is queried. The returned device information can include but is not limited to the device name, the device identifier, the device type and the online state.

[0246] The device introduction module of the device management service returns the device information to the device management module of the hardware virtualization service according to the device information query instruction if the corresponding device is queried. The returned device information can include but is not limited to the device name, the device identifier, the device type and the online state.

[0247] S309, the device management module of the hardware virtualization service in the tablet sends the device check success indication to the hardware virtualization API after receiving the device information.

[0248] The device management module of the hardware virtualization service receives the device information returned by the device introduction module of the device management service, and sends the device check success indication to the hardware virtualization API to indicate that the desk lamp to be linked succeeds in the check.

[0249] S310, the hardware virtualization API in the tablet sends a device connection request to the device management module of the hardware virtualization service.

[0250] The hardware virtualization API sends a device connection request to the device management module of the hardware virtualization service after confirming that the desk lamp to be linked succeeds in the check. The device connection request is used to indicate the establishment of a network connection with the device to be linked, and is specifically used to indicate the establishment of a network connection with the desk lamp to be linked in this embodiment. The device connection request can include but is not limited to the request type, the device name, the device identifier and the device type.

[0251] S311, the device management module of the hardware virtualization service receives the device connection request, performs a negotiation channel establishment preparation operation, and sends a negotiation channel opening request to the transmission management service.

[0252] The device management module of the hardware virtualization service receives the device connection request, prepares the negotiation channel. When the device management module prepares the negotiation channel, a Session Server is created, and the Session Name of the session service is sent to the transmission management service. After the negotiation channel is prepared, a negotiation channel opening request is sent to the transmission management service. The negotiation channel opening request is used to indicate the establishment of the negotiation channel, and the negotiation channel opening request can include but is not limited to the opposite device identifier (i.e. the desk lamp identifier) and the Session Name.

[0253] In this embodiment, the negotiation channel opening request is actively initiated by the tablet side, that is, the tablet needs to establish a connection with the desk lamp. At this time, the desk lamp can be understood as a server, and the tablet can be understood as a client that needs to access the server.

[0254] S312, the transmission management service in the tablet establishes a negotiation channel with the device interconnection service in the desk lamp.

[0255] After the transmission management service receives the negotiation channel opening request, it interacts with the device interconnection service in the desk lamp according to the Session Name to establish a negotiation channel. Specifically, the negotiation channel can be established by creating a session and determining a session identifier.

[0256] S313, the device interconnection service in the desk lamp sends a negotiation channel successful establishment indication to the camera component in the hardware abstraction service.

[0257] After the negotiation channel is established, the device interconnection service in the desk lamp sends a negotiation channel successful establishment indication to the camera component in the hardware abstraction service to indicate that the negotiation channel is established and that there is currently a device that needs to be connected. The negotiation channel successful establishment indication can include, but is not limited to, device information (i.e., tablet device information) that needs to be connected and a session identifier.

[0258] S314, the transmission management service in the tablet sends a negotiation channel successful establishment indication to the device management module of the hardware virtualization service.

[0259] After the negotiation channel is established, the transmission management service in the tablet sends a negotiation channel successful establishment indication to the device management module of the hardware virtualization service to indicate that the negotiation channel is established and that there is currently a device that needs to be connected. The negotiation channel successful establishment indication can include, but is not limited to, device information (i.e., desk lamp device information) that needs to be connected and a session identifier.

[0260] The execution order of S313 and S314 is not limited in this embodiment.

[0261] S315, the device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the desk lamp based on the negotiation channel.

[0262] The device negotiation request can include, but is not limited to, device information (such as device name, device identifier, device type, etc.) and a control channel connection request.

[0263] S316, after the camera component of the hardware abstraction service in the desk lamp receives the device negotiation request, it prepares a control channel and returns device negotiation information to the device management module of the hardware virtualization service in the tablet.

[0264] The camera component of the hardware abstraction service in the desk lamp receives the device negotiation request, parses the device negotiation request to record the device information of the opposite end, and determines an IP address and a port to be listened to according to the control channel connection request, and adds the IP address and the port to the device negotiation information and returns the device management module of the hardware virtualization service in the tablet.

[0265] It should be noted that the device negotiation request and the device negotiation information are transmitted based on the established negotiation channel.

[0266] S317, after the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the negotiation channel is closed.

[0267] The closing of the negotiation channel can be specifically closing a session. When the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, this session ends, and the device management module of the hardware virtualization service can close the corresponding session according to the session identifier.

[0268] S318, the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

[0269] The control channel opening request is used to indicate the establishment of a network communication connection with the desk lamp. The control channel opening request can include but is not limited to a communication protocol, a source IP, a source port, a destination IP and a destination port, which are the IP and port listened to by the camera component of the hardware abstraction service in the desk lamp.

[0270] S319, the tablet transmission management service connects the control channel of the camera component of the hardware abstraction service in the desk lamp, and sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet.

[0271] The tablet transmission management service receives the control channel opening request, establishes the control channel connection with the desk lamp according to the information carried by the control channel opening request, that is, establishes the network communication connection between the tablet and the desk lamp. Further, the device management module of the hardware virtualization service in the tablet and the camera component of the hardware abstraction service in the desk lamp can perform network communication based on the control channel.

[0272] After the control channel is successfully established, the tablet transmission management service sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. The control channel successful connection indication can include but is not limited to a connection success identifier and control channel related information.

[0273] S321, the device management module of the hardware virtualization service in the tablet sends a device connection success indication to the hardware virtualization API.

[0274] The device connection success indication can include, but is not limited to, a connection success identifier and device information of the connection.

[0275] S322, the hardware virtualization API in the tablet sends a device capability request to the device management module of the hardware virtualization service.

[0276] After receiving the device connection success indication, the hardware virtualization API sends a device capability request to the device management module of the hardware virtualization service. The device capability request can be used to request to obtain the virtualization capability information of the opposite device (i.e. the desk lamp). Exemplarily, the virtualization device capability information includes, but is not limited to, whether to support virtualization camera, whether to support virtualization microphone, etc.

[0277] S323, the device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the desk lamp in the control channel.

[0278] S324, the camera component of the hardware abstraction service in the desk lamp returns the device capability information to the device management module of the hardware virtualization service in the tablet in the control channel.

[0279] In this embodiment, the returned device capability information of the desk lamp at least includes the capability of supporting virtualization camera and the camera identifier of the desk lamp.

[0280] S325, the device management module of the hardware virtualization service in the tablet sends the device capability information to the hardware virtualization API.

[0281] The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API, so that the hardware virtualization API knows whether the desk lamp has the capability of supporting virtualization camera.

[0282] S326, the hardware virtualization API in the tablet sends a virtual camera enable request to the device management module of the hardware virtualization service.

[0283] The hardware virtualization API in the tablet knows that the desk lamp has the capability of supporting virtualization camera, and sends a virtual camera enable request to the device management module of the hardware virtualization service. The virtual camera enable request can include, but is not limited to, a request type and a camera identifier of the desk lamp.

[0284] S327, the device management module of the hardware virtualization service in the tablet registers a virtual camera in the virtual camera HAL.

[0285] The device management module of the hardware virtualization service receives the virtual camera enabling request, and sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request can include, but is not limited to, a request type and a camera identifier of the desk lamp. After the virtual camera HAL receives the virtual camera registration request, a virtual camera driver is registered for the camera of the desk lamp in the virtual camera HAL, and a camera ID (i.e., a virtual camera ID) is allocated to the camera of the desk lamp, and the camera ID is registered in the system. Thus, a mapping relationship between the camera of the desk lamp and the virtual camera is established in the virtual camera HAL.

[0286] S328, the device management module of the hardware virtualization service in the tablet sends a service state update indication to the camera component of the hardware abstraction service in the desk lamp.

[0287] The service state update indication is used to instruct the camera component of the hardware abstraction service in the desk lamp to update the virtualization service state thereof. The virtualization service state can include an occupied state and an unoccupied state, or can include a registered state and an unregistered state. For example, the service state update indication can include, but is not limited to, device information of the peer device (i.e., the desk lamp), a hardware identifier (such as a camera identifier of the desk lamp), and a virtualization service state corresponding to the hardware identifier.

[0288] S329, the camera component of the hardware abstraction service in the desk lamp updates the service state according to the service state update indication.

[0289] When the virtualization service state corresponding to the camera of the desk lamp in the service state update indication is indicated as the occupied state (or the registered state), the camera component updates the virtualization service state corresponding to the camera of the desk lamp as the occupied state (or the registered state).

[0290] S330, the device management module of the hardware virtualization service in the tablet sends a virtual camera enabling success indication to the hardware virtualization API.

[0291] The virtual camera enabling success indication can include, but is not limited to, an enabling success identifier (or a virtualization success identifier), a camera identifier of the desk lamp, and a camera ID corresponding to the virtual camera (or a camera ID corresponding to the camera of the desk lamp).

[0292] The execution order of S328 and S330 is not limited in this embodiment.

[0293] S331, the hardware virtualization API in the tablet sends the virtual camera enabling success indication to the education APP.

[0294] 3. Virtual camera preview access phase

[0295] S3.1, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

[0296] The virtual camera access instruction refers to an instruction for calling a virtual camera. The virtual camera access instruction can include, but is not limited to, an instruction type, a virtual camera ID, and camera configuration parameters, which include, but are not limited to, a camera resolution and a frame rate.

[0297] S3.2, the camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

[0298] After the camera service receives the virtual camera access instruction, a corresponding image preview request is generated according to the virtual camera ID and sent to the virtual camera HAL. The image preview request is used to request a preview image data stream. The image preview request can include, but is not limited to, a request identifier, a virtual camera ID, and camera configuration parameters.

[0299] S3.3, the virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0300] After the virtual camera HAL receives the image preview request, a matched virtualization hardware identifier is determined according to the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera according to the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request according to the determined virtualization hardware identifier and sends it to the hardware virtualization service. The image preview request can include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualization hardware identifier (i.e., a desk lamp camera identifier), and camera configuration parameters.

[0301] S3.4, the hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0302] The hardware virtualization service sends the image preview request to the transmission management service. The image preview request can include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualization hardware identifier (i.e., a desk lamp camera identifier), and camera configuration parameters.

[0303] When the hardware virtualization service in the tablet sends the image preview request to the transmission management service, if it is found that no data channel is established with the desk lamp, a data channel establishment request is generated and sent to the transmission management service. The data channel establishment request is used to indicate the transmission of data between the tablet and the desk lamp. The data channel establishment request can include, but is not limited to, a session identifier, connection information, and a data encoding and decoding mode.

[0304] The tablet transmits the data channel establishment request to the lamp, and the tablet transmits the data channel establishment request to the lamp. The data channel establishment request is transmitted to the lamp through the data channel. The data channel is established between the tablet and the lamp. Further, the tablet transmits various data including but not limited to image data based on the data channel between the tablet and the lamp.

[0305] After the data channel is established successfully, the tablet transmits the data channel connection success indication to the hardware virtualization service in the tablet, and the lamp transmits the data channel connection success indication to the camera component in the hardware abstraction service. The data channel connection success indication can include but is not limited to connection success identification and data channel related information.

[0306] S3.5, the tablet transmits the image preview request to the device interconnection service of the lamp.

[0307] The tablet transmits the image preview request to the device interconnection service of the lamp according to the device information carried in the image preview request.

[0308] S3.6, the device interconnection service of the lamp sends the image preview request to the camera driver.

[0309] After the device interconnection service of the lamp receives the image preview request, the corresponding hardware driver (camera driver in this embodiment) is determined according to the virtualization hardware identifier, and the corresponding image preview request is sent to the camera driver.

[0310] S3.7, the camera driver of the lamp drives the camera to collect images, and transmits the preview image data to the hardware virtualization service of the tablet through the data channel.

[0311] The camera driver drives the camera to start and collect images with the camera configuration parameters carried in the image preview request to obtain a preview image data stream, and sends the preview image data stream to the device interconnection service through the hardware abstraction service, so that the device interconnection service continuously transmits the preview image data stream to the hardware virtualization service of the tablet in the data channel. The packaging and coding process of the preview image data stream is not repeated here.

[0312] S3.8, the hardware virtualization service of the tablet transmits the preview image data to the virtual camera HAL.

[0313] The hardware virtualization service continuously receives the preview image data stream and transmits the preview image data stream to the virtual camera HAL.

[0314] S3.9, the virtual camera HAL of the tablet transmits the preview image data to the camera service.

[0315] At this time, the virtual camera HAL continuously acquires the preview image data collected by the desk lamp camera and continuously sends the preview image data to the camera service.

[0316] S3.10, the camera service in the tablet sends the preview image data to the education APP.

[0317] S3.11, the education APP in the tablet displays the preview image.

[0318] After the education APP receives the preview image data stream through the camera service, it can display the preview image in the corresponding interface.

[0319] 4. Virtual camera photographing stage

[0320] S4.1, in response to the received user operation, the education APP in the tablet sends a photographing request to the hardware virtualization service.

[0321] The user operation may be, for example, an operation of clicking a photographing option. In response to the received user operation, the education APP in the tablet sends a photographing request to the hardware virtualization service. The photographing request may include, but is not limited to, a photographing image serial number, device information (i.e., desk lamp information), virtualization hardware identification (i.e., desk lamp camera identification), and camera configuration parameters.

[0322] The photographing request may also carry a task identification to ensure the orderly management of multiple photographing tasks.

[0323] S4.2, the hardware virtualization service in the tablet sends the photographing request to the transmission management service.

[0324] S4.3, the transmission management service in the tablet transmits the photographing request to the device interconnection service of the desk lamp.

[0325] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the desk lamp in the control channel.

[0326] S4.4, the device interconnection service in the desk lamp issues a photographing request to the camera driver.

[0327] After the device interconnection service in the desk lamp receives the image preview request, it determines the corresponding hardware driver (i.e., the camera driver in this embodiment) according to the virtualization hardware identification, and sends the corresponding photographing request to the camera driver.

[0328] S4.5, the camera driver in the desk lamp drives the camera to capture an image, and transmits the captured image data to the hardware virtualization service of the tablet through a data channel.

[0329] The camera driver drives the camera to capture an image according to the camera configuration parameters carried in the photograph request, obtains photograph image data, and sends the photograph image data to the device interconnection service through the hardware abstraction service, so that the device interconnection service continuously transmits the photograph image data to the hardware virtualization service in the tablet in the data channel. The packaging and coding process of the photograph image data will not be described here.

[0330] S4.6, the hardware virtualization service in the tablet sends the photograph image data to the education APP.

[0331] S4.7, the education APP in the tablet displays the photograph image.

[0332] After the education APP receives the photograph image through the hardware virtualization service, the photograph image can be displayed in the corresponding interface.

[0333] In this embodiment, the virtual camera preview access stage is based on the original camera framework of Android, and the virtual camera photographing stage is based on the private virtualization camera framework, so that the processing path involved in the virtual camera photographing stage is shorter and the time delay of photographing is smaller. At the same time, since the image preview is also based on the original camera framework of Android, the education APP needs to make fewer changes to adapt to the technical solutions provided in this embodiment.

[0334] It should be noted that the stage division in the above process is only an exemplary expression, and the embodiment of the present application does not limit it. Moreover, after the tablet displays the preview image in the virtual camera preview access stage process, the process of displaying the preview image in real time and the process of the virtual camera photographing stage can be executed simultaneously. The details not explained in the above process can be referred to the existing technology, and will not be described here.

[0335] Figure 4a The communication architecture of the collaborative work system is given in the prior art, which is used to complete the management of the virtual camera (i.e. the desk lamp camera) by the tablet, the control command interaction between the tablet and the desk lamp, and the image data return and processing.

[0336] It should be noted that the instructions, requests and the like transmitted across devices (i.e. between the tablet and the desk lamp) need to be encapsulated based on the communication protocol and parameter sequence, and the embodiment will not be described here. The hardware virtualization service in the tablet can also manage the life cycle of the preview image stream and the photograph image by dynamically allocating and destroying memory.

[0337] In addition, it should be noted that before the collaborative work method provided in the embodiment is executed, the education APP is bound with the desk lamp, and the desk lamp is registered to the cloud server.

[0338] The embodiment of the present application provides a framework scheme for an Android system device to use a camera of an external device to take a picture, which can be applied to an education scene and is suitable for other devices provided with a camera, and the devices can share the camera capability to a mobile phone, a tablet and the like Android system device, so that the Android system device and the devices are interconnected.

[0339] Figures 5a-5b An application scenario is exemplified. As shown in (1) of Figure 5a , a tablet displays an interface 401, and a plurality of application icons are displayed on the interface 401; a user clicks an education application icon 4011. In response to the received user operation, the tablet starts the education application, and the tablet displays an education application interface, which can be referred to (2) of Figure 5a . As shown in (2) of Figure 5a , the tablet displays an education application interface 402, and a plurality of function options of the education application are displayed on the interface 402, including but not limited to a word checking function, a reading aloud function, a homework function and a picture taking function. In the case that the user uses the picture taking function, the user clicks a picture taking function option 4021, and in response to the user operation, the tablet executes the processes of a device discovery stage and a virtual camera service enabling stage and a virtual camera preview access stage.

[0340] In the device discovery stage, if the device management service of the tablet filters one desk lamp capable of being linked with the tablet, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays, for example, the interface shown in (1) of Figure 5b . In the device discovery stage, if the device management service of the tablet filters a plurality of desk lamps capable of being linked with the tablet, the tablet displays a desk lamp selection interface. Exemplarily, a list of the desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation; in response to the selection operation of the user, the education application determines one desk lamp to be linked, and continues to execute the processes of the virtual camera service enabling stage and the virtual camera preview access stage, to display, for example, the interface shown in (1) of Figure 5b .

[0341] As shown in (1) of Figure 5b , an image preview window 4031 and a picture taking option 4032 are displayed on the interface 403, and a preview image collected by the desk lamp camera in real time is displayed in the image preview window 4031. At this time, if the user clicks the picture taking option 4032, in response to the user operation, the tablet executes the process of a virtual camera picture taking stage, and displays, for example, the interface shown in (2) of Figure 5b . As shown in (2) of Figure 5b(2) As shown in the interface 404, the image captured by the lamp's camera is displayed in the image preview window 4041. If the user clicks the confirmation option 4041, the tablet responds to the user's operation, saves the captured image, and continues to display, for example... Figure 5b The preview interface shown in (1) is shown in the image. If the user clicks the cancel option 4042, the tablet will respond to the user's operation and may display, for example... Figure 5b The preview interface shown in (1) is shown in the middle.

[0342] It should be pointed out that, such as Figure 5b (2) The interface shown is merely an example. The image captured by the lamp camera may not be displayed in the image preview window 4041, but in other areas of the interface. The image preview window 4041 continues to display the preview image captured by the lamp camera in real time. This application does not limit this.

[0343] This embodiment provides a collaborative working method, primarily illustrating a low-cost technical solution for implementing online education functions based on a combination of a tablet device and a desk lamp device. The technical solution provided in this embodiment is further explained below in conjunction with several different functions involved in online education.

[0344] Scene 1

[0345] Reference Figure 1a The illustration shows an application scenario, using a word lookup function as an example to illustrate the technical solution. When students encounter unfamiliar words, they can point their finger under the word, and the tablet's camera will take a picture. The tablet will then recognize the image to identify the word and provide feedback on its meaning via the display screen after the online word lookup is completed, such as displaying and reading out the word's explanation on the interface.

[0346] like Figure 6a The diagram shows the interaction between each module. (Refer to...) Figure 6a This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0347] S501, in response to the user's click on the word search function, the tablet and desk lamp execute the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase, with the tablet displaying the preview interface.

[0348] The processes for the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0349] It should be noted that in the virtual camera preview access stage, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the camera configuration parameters are carried in the virtual camera access instruction. The camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The desk lamp camera sets according to the received configuration parameters, and acquires preview image data according to the corresponding image resolution and image acquisition frame rate.

[0350] In this scenario, the tablet needs to accurately identify the preview image to determine the content of the word pointed by the user, so the image quality requirement of the preview image in this scenario is higher, for example, the image resolution can be set to 1080P. In this way, a higher word search success rate can be ensured.

[0351] S502, the education APP in the tablet performs finger identification on the preview image.

[0352] For example, the education APP can perform finger identification on each received frame of preview image, or perform finger identification on the latest received preview image at a time, and the present embodiment does not limit this.

[0353] For example, the education APP can integrate an image recognition algorithm to realize the image recognition operation, or the education APP can call an image recognition service to perform the image recognition operation, and the present embodiment does not limit this.

[0354] The image recognition algorithm can refer to the prior art, and the present embodiment will not be repeated here.

[0355] S503, in response to the operation of the user using the finger to point to the word, the education APP in the tablet identifies the user's finger in the preview image.

[0356] When the user uses the finger to point to a certain word in the book, the desk lamp camera can acquire the preview image of the finger pointing to the word, and then the education APP in the tablet can identify the user's finger in the preview image.

[0357] When the education APP identifies the finger in the preview image, the position information of the finger in the preview image can be obtained, such as coordinate information.

[0358] S504, the education APP in the tablet determines the ROI (region of interest) image according to the position of the finger in the preview image.

[0359] After the education APP identifies the user's finger in the preview image, the ROI image can be determined according to the position information of the finger in the preview image. The education APP can determine the ROI information according to the coordinate information of the finger in the preview image, and the ROI information includes but is not limited to the center point coordinates and the area range (such as the width and height information). Further, the education APP can crop the ROI image in the preview image according to the ROI information.

[0360] S505, the education APP in the tablet accurately identifies the ROI image and determines the word to be explained.

[0361] For example, the education APP can integrate an image recognition algorithm to accurately identify the ROI image, or the education APP can call an image recognition service to accurately identify the ROI image to determine the word to be explained. The present embodiment does not limit this.

[0362] The image recognition algorithm can refer to the prior art, and the present embodiment will not be repeated here.

[0363] S506, the education APP in the tablet performs a word lookup operation on the word to be explained and displays the explanation of the word to be explained.

[0364] After determining the word to be explained, the education APP can perform an online word lookup operation or a word lookup operation in the database to obtain the explanation of the word to be explained. Further, the education APP can display the explanation of the word to be explained for the user to view. In addition, the education APP can read the displayed word explanation, and the present embodiment does not limit this.

[0365] Similarly, the user can also use a pen or other pointing tool (or indicating tool) to perform a pointing operation, and the present embodiment does not limit this. Accordingly, the education APP performs pointing tool recognition on the preview image to determine whether the user has a word lookup intention, and determines the ROI image according to the position information of the pointing tool in the preview image.

[0366] Similarly, the user can also use a finger or a pen or other pointing tool to point to a picture in the book. Accordingly, the education APP determines the ROI image according to the position information of the finger or the pen or other pointing tool, and performs picture content recognition on the ROI image, displays the corresponding explanation of the picture, and can also read the displayed picture explanation. This scenario will not be repeated here.

[0367] Figure 1a 、 Figures 7a-7b For example, an application scenario is shown. As Figure 7aAs shown in Figure (1), the tablet displays an educational app interface 701, which displays various function options of the educational application, including but not limited to word lookup, point-and-read function, homework function, and photo function. When the user clicks the word lookup function option 7011, in response to the user's operation, the tablet executes the device discovery phase, the virtual camera service enable phase, and the virtual camera preview access phase.

[0368] During the device discovery phase, if the tablet's device management service filters out only one desk lamp that can be linked with the tablet, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase, and displays, for example... Figure 7a The interface shown in (2) is as follows. During the device discovery phase, if the number of desk lamps that can be linked with the tablet is multiple as filtered by the tablet's device management service, the tablet displays a desk lamp selection interface. For example, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked and continues to execute the virtual camera service enable phase and the virtual camera preview access phase to display, for example... Figure 7a The interface shown in (2) is shown in the middle.

[0369] like Figure 7a As shown in Figure (2), interface 702 displays an image preview window 7021 and a word lookup function operation diagram 7022. The image preview window 7021 displays a preview image captured in real-time by the desk lamp's camera. Users can refer to the word lookup function operation diagram 7022 to perform a pointing-to-word or pointing-to-image operation to trigger the word lookup function. The educational app performs pointing-to-word or pointing-to-image recognition based on the preview image. (Continue referring to...) Figure 7a In section (2), when a user points to a word in a book, the desk lamp camera captures a preview image of the finger pointing to the word, which is displayed in the image preview window 7021. Furthermore, the educational app can identify the user's finger in the preview image and determine the finger's position information in the preview image, such as coordinates. The educational app determines the ROI image based on the finger's position in the preview image and performs accurate recognition of the ROI image to identify the new word to be explained. After the educational app finds the definition of the new word, it displays the corresponding definition on the interface for reference. Figure 7b As shown.

[0370] However, in the above process, to ensure a high success rate for word lookup, the desk lamp's camera needs to continuously return a high-resolution (e.g., 1080p) preview image stream, which places higher demands on bandwidth, requiring 4-8 Mbps. This also increases the hardware requirements for the desk lamp, thereby raising its cost.

[0371] To achieve the scheme of the tablet and the desk lamp cooperating to realize the word searching function, and reduce the hardware cost of the desk lamp, the embodiment further provides a technical scheme. Since the requirements of the finger recognition or the finger word tool recognition and the position recognition on the image resolution are not high, in the virtual camera preview access stage, the desk lamp camera continuously returns a low-resolution (for example, 480p) preview image stream. If the education APP recognizes the finger word or the finger image operation of the user based on the preview image, the desk lamp captures a high-resolution (for example, 1080p) image, so that the education APP accurately identifies and determines the word or the image to be explained based on the high-resolution image. In this way, the requirement of the bandwidth is not high when the desk lamp camera continuously returns the low-resolution preview image stream, and only 0.5-1 Mbps of bandwidth is required. Only when the high-resolution image is transmitted, more bandwidth is required. Therefore, the technical scheme not only reduces the requirement of the hardware chip of the desk lamp and reduces the cost of the desk lamp, but also guarantees the word searching success rate.

[0372] As shown in Figure 6b is an interaction schematic diagram of each module. Referring to Figure 6b , the embodiment of the application provides a method flow of the tablet and the desk lamp cooperating, specifically including:

[0373] S601, in response to the operation of the user clicking the word searching function, the tablet and the desk lamp execute the flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage, and the tablet displays a preview interface.

[0374] The flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage can be referred to the foregoing, and will not be described here.

[0375] It should be noted that in the virtual camera preview access stage, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the camera configuration parameters are carried in the virtual camera access instruction. The camera configuration parameters can include but are not limited to the image resolution and the image acquisition frame rate. The desk lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the corresponding image resolution and the image acquisition frame rate.

[0376] In this scenario, in order to reduce the bandwidth occupied by the preview image stream, in the virtual camera preview access stage, the desk lamp camera can collect the preview image according to the low resolution. For example, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the first configuration parameters are carried in the virtual camera access instruction. The first configuration parameters include the first image resolution (such as 480P). Further, the desk lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the first image resolution and the corresponding image acquisition frame rate.

[0377] S602, the tablet-side education APP performs finger recognition on the preview image.

[0378] S603, in response to the user's operation of using the finger to point at the word, the tablet-side education APP identifies the user's finger in the preview image.

[0379] S604, the tablet-side education APP determines the ROI information according to the position of the finger in the preview image, and generates a photograph request according to the ROI information.

[0380] The ROI information refers to information used to determine the ROI, and can include but is not limited to the center point coordinates and the area range (such as the width and height information).

[0381] S605, the tablet transmits the photograph request to the desk lamp side.

[0382] The photograph request can include but is not limited to a virtual camera ID corresponding to the desk lamp camera, a second configuration parameter of the desk lamp camera, and the ROI information. The second configuration parameter includes but is not limited to a second image resolution, which is higher than the first image resolution, such as 1080P. In this way, the education APP can accurately identify and determine the word or picture to be explained according to the high-resolution image.

[0383] S606, the camera of the desk lamp is set according to the second configuration parameter carried in the photograph request, an image is captured according to the second image resolution, and the captured image is sent to the hardware abstraction service.

[0384] S607, the hardware abstraction service determines the ROI image according to the ROI information.

[0385] The hardware abstraction service can crop the ROI image from the captured image according to the ROI information.

[0386] In an optional embodiment, the photograph request includes but is not limited to the second configuration parameter, but does not include the ROI information. In this way, the tablet transmits the photograph request to the desk lamp side, the camera of the desk lamp is set according to the second configuration parameter carried in the photograph request, an image is captured according to the second image resolution, and the captured image is returned to the tablet-side education APP. Further, the education APP can determine the ROI image according to the ROI information, for example, crop the ROI image from the captured image according to the ROI information.

[0387] S608, the desk lamp transmits the ROI image to the tablet-side education APP.

[0388] Compared with the desk lamp directly returning the high-resolution captured image to the tablet-side education APP, returning the cropped ROI image to the tablet-side education APP can also reduce the data transmission amount and reduce the bandwidth occupation.

[0389] S609, the education APP in the tablet accurately identifies the ROI image to determine the word to be explained.

[0390] S610, the education APP in the tablet performs a word search operation on the word to be explained, and displays the explanation of the word to be explained.

[0391] The details of the process not explained in detail can be referred to the foregoing, which will not be repeated here.

[0392] Similarly, the user can also use a pointing pen or other pointing tool to perform a pointing operation. The present embodiment does not limit this. Correspondingly, the education APP identifies the pointing tool in the preview image to determine whether the user has a word search intention, and determines the ROI information according to the position information of the pointing tool in the preview image.

[0393] Similarly, the user can also use a finger or a pointing pen or other pointing tool to point to a picture in the book. Correspondingly, the education APP determines the ROI information according to the position information of the finger or the pointing pen or other pointing tool, and determines the ROI image of the ROI information in the photographed image, performs picture content recognition on the ROI, displays the explanation corresponding to the picture, and can also read the displayed picture explanation. As for this case, the present embodiment will not be repeated.

[0394] As for the application scenarios of this process, the application scenarios shown in Figure 1a 、 Figures 7a-7b can be referred to. Referring to (2) in Figure 7a , when the user points to a word in the book with a finger, the finger pointing preview image is captured by the desk lamp camera, and the finger pointing preview image is displayed in the image preview window 7021. Further, the education APP can identify the user's finger in the preview image, determine the ROI information, and generate a photographing request according to the ROI information and the high-resolution image, to trigger the desk lamp camera to photograph the image. The desk lamp camera photographs the image according to the high-resolution image, and the desk lamp side crops the high-resolution photographed image to obtain the ROI image according to the ROI information, and returns the ROI image to the education APP in the tablet. The education APP accurately identifies the ROI image to determine the word to be explained. After the education APP queries the word explanation, the corresponding word explanation is displayed on the interface, which can be referred to Figure 7b .

[0395] Scenario Two

[0396] Referring to the application scenario diagram shown in Figure 1a , the present scenario takes the homework function as an example to describe the technical solution. When a student needs to submit homework online, he or she can click on the photograph in the education APP, use the desk lamp camera to photograph the homework image, and upload the homework image to the database through the education APP.

[0397] As Figure 8 shown is an interaction diagram of each module. Referring to Figure 8 , the embodiment of the application provides a method flow for cooperation of a tablet and a desk lamp, and specifically comprises the following steps:

[0398] S801, in response to an operation of a user clicking a homework function, an education APP in the tablet displays a homework submission list.

[0399] It should be noted that the homework submission list refers to a list comprising a plurality of homework submission options, and one homework submission option corresponds to one homework, which can be referred to the interface 704 shown in Figure 9a . Wherein, the homework option can be divided according to subjects or time, and the embodiment does not limit this.

[0400] If only one homework image needs to be submitted in the homework function of the education APP, the education APP does not display the homework submission list. At this time, in response to the operation of the user clicking the homework function, the tablet and the desk lamp execute the flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage, and the tablet displays a preview interface.

[0401] S802, in response to the user clicking the homework submission option, the tablet and the desk lamp execute the flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage, and the tablet displays the preview interface.

[0402] Wherein, the flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage can be referred to the foregoing, and will not be described here.

[0403] It should be noted that in the virtual camera preview access stage, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the camera configuration parameters are carried in the virtual camera access instruction. Wherein, the camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The desk lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the corresponding image resolution and image acquisition frame rate.

[0404] In this scenario, in order to reduce the bandwidth occupied by the preview image stream, in the virtual camera preview access stage, the desk lamp camera can collect the preview image according to the low resolution. For example, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the first configuration parameters are carried in the virtual camera access instruction, wherein the first configuration parameters include the first image resolution (such as 480P). Further, the desk lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the first image resolution and the corresponding image acquisition frame rate.

[0405] S803, in response to the user clicking the photographing option, the tablet generates a photographing request.

[0406] When the user places the homework or book in the camera capture area of the desk lamp, the user can click the photographing option to trigger the desk lamp camera to capture the homework image.

[0407] The photographing request can include but is not limited to the virtual camera ID corresponding to the desk lamp camera and the second configuration parameter of the desk lamp camera. The second configuration parameter includes but is not limited to the second image resolution, which is higher than the first image resolution, such as the second image resolution is set to 1080P. In this way, the education APP can upload high-resolution homework images.

[0408] S804, the tablet transmits the photographing request to the desk lamp.

[0409] S805, the camera of the desk lamp sets the second configuration parameter carried in the photographing request and captures the image according to the second image resolution.

[0410] S806, the desk lamp transmits the captured image to the education APP in the tablet.

[0411] S807, the education APP in the tablet displays the captured image.

[0412] The education APP in the tablet receives the homework image captured by the desk lamp camera and displays the homework image. If the user is satisfied with the captured homework image, the user can click the submission option to upload the homework image to the database. If the user is not satisfied with the captured homework image, the user can click the photographing option again to trigger the desk lamp camera to capture the homework image again.

[0413] S808, in response to the user clicking the submission option, the education APP in the tablet uploads the captured image to the database.

[0414] The details of the above process are not explained in detail and are not described here.

[0415] It should be noted that the above-mentioned homework image is only an example and the user can click the photographing option to trigger the desk lamp camera to capture other images. After the desk lamp transmits the captured image to the education APP in the tablet, the education APP can upload the received captured image to the corresponding database.

[0416] Figure 1a 、 9a - Figure 9c An example application scenario is shown. For example, Figure 9aAs shown in FIG. 9b (1), the tablet displays an education APP interface 701. The interface 701 displays various function options of the education application, including but not limited to a word search function, a read-aloud function, a homework function, a photographing function, etc. The user clicks the word search function option 7012. In response to the user operation, the tablet displays a homework submission list, which can refer to Figure 9a As shown in FIG. 9b (2). Figure 9b As shown in the homework submission list interface 704 in FIG. 9b (2), a plurality of homework submission options (e.g., submission 1, submission 2, submission 3, submission 4, etc.) are displayed. Different homework submission options correspond to different homework. Taking the user’s need to upload a homework image for submission 4 as an example, the user clicks the submission 4 option 7042. In response to the user operation, the tablet can display an interface 705 as shown in FIG. 9b (1). Figure 9b As shown in FIG. 9b (1), the tablet displays an education APP interface 701. The interface 701 displays various function options of the education application, including but not limited to a word search function, a read-aloud function, a homework function, a photographing function, etc. The user clicks the word search function option 7012. In response to the user operation, the tablet displays a homework submission list, which can refer to

[0417] Continuing to refer to FIG. 9b (1), in the homework submission interface 705, an image preview window 7041, a photographing option 7051, and a submission option 7052 are displayed. The image preview window 7041 displays a preview image captured by the desk lamp camera in real time. When the user places a homework or a book in the capture area of the desk lamp camera, the user can click the photographing option 7051 to trigger the desk lamp camera to capture a homework image. In response to the user operation, the education APP generates a photographing request and sends it to the desk lamp to call the desk lamp camera to capture the homework image. The desk lamp camera captures the homework image at a high resolution according to the image carried in the photographing request and returns the captured homework image to the education APP in the tablet for display, which can refer to the interface 706 as shown in FIG. 9b (2). Figure 9c As shown in FIG. 9b (2).

[0418] Continuing to refer to FIG. 9b (2), the interface 706 displays the image preview window 7041, the photographing option 7051, the submission option 7052, and the homework image 7061 captured by the desk lamp camera. The homework image 7061 further displays a close option 7062. If the user is not satisfied with the homework image 7061, the user can click the close option 7062, and the homework image 7061 will no longer be displayed on the interface. At this time, the user can click the photographing option 7051 to trigger the desk lamp camera to capture a homework image again. If the user is satisfied with the homework image 7061, the user can click the submission option 7052. In response to the user operation, the tablet can display an interface 707 as shown in FIG. 9b (3). Figure 9bThe to-be-confirmed interface 701 is shown. In the to-be-confirmed interface 701, the to-be-submitted work image 7061 is displayed, and a confirmation submission window 7071 is displayed. If the user clicks the confirmation option 7072 in the confirmation submission window 7071, the educational APP responds to the user operation to upload the work image 7061 to the database. If the user clicks the cancel option 7071 in the confirmation submission window 7071, the educational APP can respond to the user operation to display the interface as shown in (1) of FIG. 9b, so as to wait for the user to click the photographing option 7051 to trigger the table lamp camera to re-shoot the work image. Figure 9c It is pointed out that, still referring to (2) of FIG. 9b, if the user is satisfied with the work image 7061, the submit option 7052 can be clicked. In response to the user operation, the educational APP can no longer display the interface as shown in (1) of FIG. 9b, and directly upload the work image 7061 to the database. The present embodiment is not limited in this regard.

[0419] It is pointed out that, still referring to (2) of FIG. 9b, if the user is satisfied with the work image 7061, the submit option 7052 can be clicked. In response to the user operation, the educational APP can no longer display the interface as shown in (1) of FIG. 9b, and directly upload the work image 7061 to the database. The present embodiment is not limited in this regard. Figure 1a

[0420] Scenario three

[0421] Referring to the application scenario diagram as shown in (1) of FIG. 9b, the present scenario takes the point reading function (or called the finger reading function) as an example to describe the technical solutions. When the student needs the educational APP to read the content in the book, the table lamp camera can be used to collect the book image in real time, and then the educational APP can load the corresponding book content according to the book image, and determine the book content to be read according to the student's finger position or page turning operation to read. Figure 10 As shown in (1) of FIG. 9b, the interaction diagram of each module is shown. Referring to (1) of FIG. 9b, the present application embodiment provides a method flow of the cooperation between the tablet and the table lamp, which specifically includes:

[0422] Figure 10 The present application embodiment provides a method flow of the cooperation between the tablet and the table lamp, which specifically includes: Figure 1a S901, in response to the user's operation of clicking the point reading function, the tablet and the table lamp execute the flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage, and the tablet displays a preview interface.

[0423] The flow of the device discovery stage, the virtual camera enabling stage and the virtual camera preview access stage can be referred to the foregoing, and will not be described herein.

[0424] It is pointed out that, in the virtual camera preview access stage, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the camera configuration parameters are carried in the virtual camera access instruction. The camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The table lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the corresponding image resolution and image acquisition frame rate.

[0425] It is pointed out that, in the virtual camera preview access stage, the hardware virtualization API in the tablet sends the virtual camera access instruction to the camera service, and the camera configuration parameters are carried in the virtual camera access instruction. The camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The table lamp camera is set according to the received configuration parameters, and the preview image data is collected according to the corresponding image resolution and image acquisition frame rate.​​

[0426] In the present scenario, in order to reduce the bandwidth occupied by the preview image stream, in the camera preview access stage, the desk lamp camera can collect the preview image in a low resolution. For example, the hardware virtualization API in the tablet sends the first configuration parameter in the virtual camera access instruction to the camera service, wherein the first configuration parameter includes the first image resolution (such as 480P). Further, the desk lamp camera is set according to the received configuration parameter, and the preview image data is collected according to the first image resolution and the corresponding image collection frame rate.

[0427] S902, the education APP in the tablet identifies the preview image and determines the book name.

[0428] For example, the education APP can identify the book information for each received frame of the preview image, or can identify the book information for the latest received preview image at a time, and the present embodiment does not limit this.

[0429] For example, the education APP can integrate an image recognition algorithm to realize the image recognition operation, or can call an image recognition service to perform the image recognition operation, and the present embodiment does not limit this.

[0430] The image recognition algorithm can refer to the prior art, and the present embodiment will not be repeated here.

[0431] S903, the education APP in the tablet retrieves the database according to the book name and loads the book content corresponding to the book name.

[0432] If the education APP in the tablet retrieves the database according to the book name and determines different versions of the book, the corresponding book list can be displayed for the user to select. Further, in response to the user's selection operation on a version of the book, the education APP loads the content corresponding to the version of the book

[0433] S904, in response to the user's page turning or finger reading operation, the education APP in the tablet identifies the preview image and determines the paragraph to be read, and reads the corresponding paragraph.

[0434] After loading the book content, the education APP can identify the pointing tool of the finger or the pen for each received frame of the preview image, or can identify the pointing tool of the finger or the pen for the latest received preview image at a time, and the present embodiment does not limit this. In response to the user's reading operation, the education APP can determine the paragraph to be read in the loaded book content according to the book page number and the user's click position information, such as coordinate information, and then read the corresponding paragraph.

[0435] After the loading of the book content is completed, the educational APP can also perform page turning recognition according to the preview image stream. In response to a page turning operation of the user, the educational APP can determine the paragraph to be read according to the recognized book page number in the loaded book content, and read the corresponding paragraph.

[0436] The part not explained in detail in the present process can refer to the foregoing, which will not be described here.

[0437] Figures 11a-11b 、 Figure 11a An exemplary application scenario is shown. As shown in (1) of Figure 11a , the tablet display educational APP interface 701, and the interface 701 displays various function options of the educational application, including but not limited to the word search function, the read-aloud function, the homework function, the photographing function, etc. The user clicks the read-aloud function option 7013, and in response to the user operation, the tablet executes the device discovery phase and the virtual camera service enabling phase, and the virtual camera preview access phase.

[0438] Among them, in the device discovery phase, if the number of table lamps capable of being linked with the tablet filtered by the device management service of the tablet is one, the tablet automatically executes the process of the virtual camera service enabling phase and the virtual camera preview access phase, and displays an interface such as (2) of Figure 11a . In the device discovery phase, if the number of table lamps capable of being linked with the tablet filtered by the device management service of the tablet is more than one, the tablet displays a table lamp selection interface. Exemplarily, the table lamp selection interface displays a list of table lamps to be linked, and the user can perform a selection operation. In response to the selection operation of the user, the educational application determines a table lamp to be linked, and continues to execute the process of the virtual camera service enabling phase and the virtual camera preview access phase to display an interface such as (2) of Figure 11a .

[0439] As shown in (2) of Figure 11b , the image preview window 7081 is displayed in the interface 708. Among them, the preview image captured by the table lamp camera in real time is displayed in the image preview window 7081. The educational APP identifies the preview image to determine the book name. After the educational APP identifies the book name, the educational APP retrieves the database according to the book name. If the corresponding book is retrieved, the loading operation of the book content is performed, which can refer to (1) of Figure 11b .

[0440] Continue to refer to Figure 11bAs shown in (1), the interface 709 displays an image preview window 7081, a recognized book name 7091, and a book content loading progress identifier 7092. After the book content is loaded, the educational APP can recognize the user's pointing operation or page turning operation according to the preview image. Taking the user's page turning operation as an example, refer to the interface 710 shown in (2). The image preview window 7081 can display the user's page turning action. In response to the user's operation, the educational APP recognizes the book page number according to the preview image. Further, the educational APP can determine the paragraph to be read in the loaded book content according to the recognized book page number, and read the corresponding paragraph. ​

[0441] In the collaborative working method provided in the embodiments of the present application, a professional online education experience can be composed by using a household tablet and a desk lamp with a camera.

[0442] The embodiments of the present application further provide a computer storage medium, which stores computer instructions. When the computer instructions run on an electronic device, the electronic device executes the related method steps to implement the collaborative working method in the embodiments.

[0443] The embodiments of the present application further provide a computer program product, which, when running on a computer, causes the computer to execute the related steps to implement the collaborative working method in the embodiments.

[0444] In addition, the embodiments of the present application further provide a device, which can be a chip, a component or a module. The device can include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the collaborative working method in the method embodiments.

[0445] The electronic device (such as a tablet, a mobile phone, an IOT device, etc.), the computer storage medium, the computer program product or the chip provided in the embodiments can be used to execute the corresponding method provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described herein again.

[0446] Through the above description of the embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. ​

[0447] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the division of the apparatus embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different apparatuses can be indirect couplings or communication connections through some interfaces, apparatuses or units, and can be in electrical, mechanical or other forms.

[0448] The above description is merely illustrative of the technical solutions of the present application, and is not a limitation on the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A system for working in cooperation, characterized by The system comprises: an electronic device and an Internet of Things (IoT) device, the IoT device being provided with a camera for capturing image data downward, and a target application in the electronic device being bound to the IoT device; the electronic device is configured to register a virtual camera corresponding to the camera in a system, send a first indication to the IoT device, and send a second indication to the target application, the second indication being used to indicate that the virtual camera is enabled successfully; the IoT device is configured to update a virtualization service state corresponding to the camera to an occupied state according to the first indication; the electronic device is further configured to send an image preview request to the IoT device by calling the virtual camera; the IoT device is configured to capture preview image data by calling the camera according to the image preview request of the electronic device, and send the preview image data to the target application in the electronic device for preview display; the electronic device is further configured to generate a photographing request when the virtual camera is called, and send the photographing request to the IoT device; the IoT device is further configured to capture an image by calling the camera according to the photographing request of the electronic device, and send the captured image data to the target application in the electronic device for display.

2. The system according to claim 1, wherein the electronic device is further configured to verify the IoT device, connect the IoT device, and obtain whether the IoT device has the capability of supporting the virtual camera before registering the virtual camera corresponding to the camera in the system; the electronic device is further configured to register the virtual camera corresponding to the camera in the system when it is determined that the IoT device has the capability of supporting the virtual camera.

3. The system for coordinated work of claim 1, wherein, The electronic device comprises a camera service, a virtual camera HAL, a hardware virtualization service, and a transmission management service; the IoT device comprises a device interconnection service; the transmission management service and the device interconnection service are used to provide the capability of interconnecting the electronic device and the IoT device; the electronic device is configured to transmit the image preview request to the device interconnection service of the IoT device through the camera service, the virtual camera HAL, the hardware virtualization service, and the transmission management service in sequence, and transmit the received preview image data to the target application for display through the transmission management service, the hardware virtualization service, the virtual camera HAL, and the camera service in sequence; wherein the target application is adapted to the hardware virtualization service and the transmission management service.

4. The system according to claim 3, wherein the electronic device is configured to transmit the photographing request to the device interconnection service of the IoT device through the hardware virtualization service and the transmission management service in sequence, and transmit the received captured image data to the target application for display through the transmission management service and the hardware virtualization service in sequence.

5. The co-working system of any of claims 1-4, wherein, The IoT device is a table lamp.

6. The system according to claim 5, wherein The electronic device is configured to: in response to a first operation, register a virtual camera corresponding to the camera in the system, the first operation being used to start a word searching function in a target application; and the target application is authenticated by the electronic device; The electronic device is further configured to: perform image recognition on a preview image, determine position information of a finger or a word pointing tool in the preview image, and determine a ROI image in the preview image according to the position information; and the word pointing tool is used to point to a word or a picture in a book; The electronic device is further configured to: perform image recognition on the ROI image, determine a word to be explained or a picture to be explained, and display an explanation corresponding to the word to be explained or the picture to be explained.

7. The collaborative working system of claim 5, wherein The electronic device is configured to: in response to a second operation, register a virtual camera corresponding to the camera in the system, the second operation being used to start a homework submission function in a target application; and the target application is authenticated by the electronic device; The electronic device is further configured to: in response to a third operation, generate a photographing request when the virtual camera is called, send the photographing request to the Internet of Things device, and display a captured image returned by the Internet of Things device; The electronic device is further configured to: in response to a fourth operation, upload the captured image to a database.

8. The collaborative working system of claim 5, wherein The electronic device is configured to: in response to a fifth operation, register a virtual camera corresponding to the camera in the system, the fifth operation being used to start a read-aloud function in a target application; and the target application is authenticated by the electronic device; The electronic device is further configured to: perform image recognition on a preview image, determine a book name, and load book content according to the book name; The electronic device is further configured to: perform image recognition on a preview image, determine position information of a finger or a word pointing tool in the preview image and a book page number, determine first content in the book content according to the book page number and the position information, and read the first content aloud; or perform image recognition on a preview image, determine a book page number, determine second content in the book content according to the book page number, and read the second content aloud.

9. The co-working system of any of claims 6-8, wherein, The electronic device includes a mobile phone and a tablet computer.

10. A method of working in concert, characterized by, The electronic device registers a virtual camera corresponding to a camera in an Internet of Things device in a system, sends a first indication to the Internet of Things device, and sends a second indication to a target application, the second indication indicating that the virtual camera is enabled successfully; wherein the target application in the electronic device is bound to the Internet of Things device; The Internet of Things device updates a virtualization service state corresponding to the camera to an occupied state according to the first indication; The electronic device sends an image preview request to the Internet of Things device by calling the virtual camera; The Internet of Things device collects preview image data by calling the camera according to the image preview request of the electronic device, and sends the preview image data to the target application of the electronic device for preview display; ​ In response to a user operation, the electronic device generates a photographing request when invoking the virtual camera, and sends the photographing request to the Internet of Things device; The Internet of Things device invokes the camera to take an image according to the photographing request of the electronic device, and sends the taken image data to the target application of the electronic device for display.

11. The method of claim 10, wherein, Before the electronic device registers the virtual camera corresponding to the camera in the system, further comprising: The electronic device verifies the Internet of Things device, connects the Internet of Things device, and obtains whether the Internet of Things device has the capability to support virtualized cameras; The electronic device registers the virtual camera corresponding to the camera in the system, comprising: The electronic device registers the virtual camera corresponding to the camera in the system when it is determined that the Internet of Things device has the capability to support virtualized cameras.

12. The method of claim 10, wherein, The electronic device includes a camera service, a virtual camera HAL, a hardware virtualization service, and a transmission management service; the Internet of Things device includes a device interconnection service; the transmission management service and the device interconnection service are used to provide the capability of interconnection between the electronic device and the Internet of Things device; The electronic device sends an image preview request to the Internet of Things device, comprising: The electronic device transmits the image preview request to the device interconnection service of the Internet of Things device through the camera service, the virtual camera HAL, the hardware virtualization service, and the transmission management service in sequence; The electronic device receives the preview image data returned by the Internet of Things device and displays it in the target application, comprising: The received preview image data is transmitted to the target application for display through the transmission management service, the hardware virtualization service, the virtual camera HAL, and the camera service in sequence.

13. The method of claim 12, wherein, The electronic device sends a photographing request to the Internet of Things device, comprising: The electronic device transmits the photographing request to the device interconnection service of the Internet of Things device through the hardware virtualization service and the transmission management service in sequence; The electronic device receives the taken image data returned by the Internet of Things device and displays it in the target application, comprising: The electronic device transmits the received taken image data to the target application for display through the transmission management service and the hardware virtualization service in sequence.

14. The method according to any one of claims 10 to 13, characterized in that, The Internet of Things device is a table lamp.

15. The method of claim 14, wherein, Further comprising: The electronic device registers the virtual camera corresponding to the camera in the system in response to a first operation, the first operation being used to start the word search function in the target application; The target application is authenticated by the electronic device; The electronic device performs image recognition on the preview image, determines the position information of a finger or a word pointing tool in the preview image, and determines the ROI image in the preview image according to the position information; the word pointing tool is used to point to the text or pictures in a book; The electronic device performs image recognition on the ROI image, determines the text to be explained or the picture to be explained, and displays the interpretation corresponding to the text to be explained or the picture to be explained.

16. The method of claim 14, wherein, Further comprising: The electronic device registers a virtual camera corresponding to the camera in the system in response to a second operation for starting a submission job function in a target application; The target application is authenticated by the electronic device; The electronic device generates a photographing request when invoking the virtual camera in response to a third operation, sends the photographing request to the Internet of Things device, and displays a captured image returned by the Internet of Things device; The electronic device uploads the captured image to a database in response to a fourth operation.

17. The method of claim 14, wherein, Further comprising: The electronic device registers a virtual camera corresponding to the camera in the system in response to a fifth operation for starting a point reading function in a target application; The target application is authenticated by the electronic device; The electronic device performs image recognition on the preview image, determines a book name, and loads book content according to the book name; The electronic device performs image recognition on the preview image, determines position information of a finger or a finger word tool in the preview image and a book page number, determines first content in the book content according to the book page number and the position information, and reads the first content aloud; or the electronic device performs image recognition on the preview image, determines a book page number, determines second content in the book content according to the book page number, and reads the second content aloud.

18. The method according to any one of claims 15-17, characterized by, The electronic device includes a mobile phone and a tablet computer.

19. An electronic device, comprising: Comprising: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, when the program instructions are executed by the processor, the electronic device executes the method steps executed by the electronic device in any one of claims 10-18 to constitute a cooperative working method.

20. An Internet of Things device comprising: Comprising: a camera, a memory and a processor, the memory being coupled to the processor, and the camera being coupled to the processor; The memory stores program instructions, when the program instructions are executed by the processor, the Internet of Things device executes the method steps executed by the Internet of Things device in any one of claims 10-18 to constitute a cooperative working method.

21. A computer readable storage medium comprising a computer program, characterized in that, When the computer program runs on the electronic device, the electronic device executes the method steps executed by the electronic device in any one of claims 10-18 to constitute a cooperative working method, or the electronic device executes the method steps executed by the Internet of Things device in any one of claims 10-18 to constitute a cooperative working method.

Images