Photographing method, electronic device and collaborative working system

Abstract

A photographing method, electronic device, and collaborative working system relate to the fields of terminal technology and the Internet of Things technology. The method comprises: sequentially receiving i photographing requests, wherein the i photographing requests are issued by j applications, each application issuing at least one photographing request, i and j being integers greater than or equal to 1, with i greater than or equal to j; sequentially inserting the photographing requests into a photographing request queue in the order in which the i photographing requests are received, with the first photographing request in the photographing request queue being the first photographing request received; sending the nth photographing request to an Internet of Things device to obtain photographing data, and upon completing the photographing task of the nth photographing request, deleting the nth photographing request from the photographing request queue until there are no photographing requests in the photographing request queue, where n is 1, 2, ..., i. Utilizing this method, an Internet of Things device can simultaneously receive multiple photographing requests and correctly transmit the photographic data back to multiple electronic devices, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a photographing method, electronic equipment, and a collaborative working system. Background Art

[0002] At present, by installing components such as the device virtualization software development kit (DVSDK) and the distributed mobile sensing development platform (DMSDP) on central-side devices such as mobile phones and tablets, and setting up corresponding DMSDP for adaptation on Internet of Things (IoT) devices that can use cameras, such as desk lamps, the central-side devices can be connected to IoT devices and the virtualized camera function of the IoT devices can be used on the central-side devices.

[0003] A typical application scenario for these technologies is education. An educational application installed on a central device can initiate a photo request. IoT devices like desk lamps then use their cameras to take photos and transmit them back to the central device for functions like answering questions and submitting homework.

[0004] However, in the current solution, once an IoT device establishes a transmission channel with a central device, it can only process a single photo request from a single application on the central device. If multiple applications on the central device simultaneously send photo requests to the IoT device, photo data transmission errors may occur, or even the virtual camera's photo service may be interrupted, impacting the user experience. Summary of the Invention

[0005] In order to solve the above problems, the present application provides a photo-taking method, an electronic device and a collaborative working system, which can enable the IoT device to correctly transmit photo data back to the corresponding application when it receives photo-taking requests sent by multiple applications on the electronic device, thereby improving the user experience.

[0006] In a first aspect, the present application provides a photographing method, which is applied to an electronic device, that is, a central-side device, and the electronic device is used to take photographs through a camera of an Internet of Things device to obtain photo data. The method includes: receiving i photographing requests in sequence, the i photographing requests are issued by j applications, each application issues at least one photographing request, i and j are integers greater than or equal to 1, and i is greater than or equal to j; inserting the photographing requests into a photographing request queue in the order in which the i photographing requests are received, and the first photographing request in the photographing request queue is the first photographing request received; sending the nth photographing request to the Internet of Things device to obtain photographing data, and when the photographing task of the nth photographing request is completed, deleting the nth photographing request from the photographing request queue until there are no photographing requests in the photographing request queue, and n is 1, 2,...i in sequence.

[0007] The solution provided by the present application utilizes a photo request queue to arrange various photo tasks, and the photo request queue is executed in a first-in-first-out and last-in-last-out order. The photo tasks in the photo request queue are completed in sequence to avoid sending multiple tasks to the IoT device side at the same time, which may cause the IoT device side to crash or return photo data errors. Completing photo tasks one by one means: sending the photo request at the head of the photo request queue to the IoT device, and after receiving the photo data returned by the IoT device, returning the photo data to the correct corresponding application. At this time, the photo task at the head of the photo request queue is considered completed. Then, the head photo task of the photo request queue is deleted. At this time, the second photo task of the original queue is updated to become the head photo task, and only then will the new head photo task be sent to the IoT device side. In summary, by using this method, by completing the photo tasks one by one, the IoT device can correctly return photos to the corresponding application when it receives photo requests sent by multiple applications on the electronic device, thereby improving the user experience.

[0008] In a possible implementation, inserting the i photo requests into the photo request queue in the order in which the i photo requests are received specifically includes:

[0009] Creating the photo-taking queue;

[0010] The photo requests are inserted into the photo request queue in sequence according to the order in which the i photo requests are received, and the correspondence between the photo request code, time information and request queue number is saved. The photo request code is the code of the photo request issued by the application itself, and the request queue number is the sequence number of each photo request in the photo request queue.

[0011] In a possible implementation, creating the photo-taking queue specifically includes:

[0012] When a transmission channel for the photo data is not established between the electronic device and the IoT device, establishing the transmission channel and creating the photo queue;

[0013] When the transmission channel has been established between the electronic device and the Internet of Things device, the photo-taking queue is created.

[0014] In a possible implementation, the photographing request code is cyclically counted between a first preset value and a second preset value.

[0015] In a possible implementation, the request queue number is cyclically counted between a third preset value and a fourth preset value.

[0016] In a possible implementation, the time information includes:

[0017] At least one of the time it takes to complete the photo-taking task or the time it takes to receive the photo-taking request.

[0018] In a possible implementation, the method further includes:

[0019] When the transmission channel between the electronic device and the IoT device is destroyed, the photo request queue is deleted to free up space and reduce resource usage.

[0020] In a possible implementation, sending an nth photo request to the IoT device to obtain photo data, and when a photo task of the nth photo request is completed, deleting the nth photo request from the photo request queue, specifically includes:

[0021] Sending an nth photo-taking request to the IoT device through the transmission channel to obtain photo-taking data;

[0022] Returning the photographing data to the application sending the nth photographing request according to the correspondence between the application's callback function and the photographing request code, so as to complete the photographing task of the nth photographing request;

[0023] The correspondence between the photographing request code of the nth photographing request, the time information of the nth photographing request and the request queue number of the nth photographing request is deleted from the photographing request queue.

[0024] In a second aspect, the present application also provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the photographing method consisting of the method steps executed by the electronic device in the second aspect and any one of the implementations of the second aspect.

[0025] The electronic device uses a photo request queue to arrange various photo tasks, and the photo request queue is executed in the order of first-in-first-out and last-in-last-out. The photo tasks in the photo request queue are completed in sequence to avoid sending multiple tasks to the IoT device side at the same time, which may cause a crash on the IoT device side or an error in the photo data being sent back. Completing the photo tasks one by one means: sending the photo request at the head of the photo request queue to the IoT device, and after receiving the photo data sent back by the IoT device, sending the photo data back to the correct corresponding application. At this time, it is considered that the photo task at the head of the photo request queue is completed. Then the head photo task of the photo request queue is deleted. At this time, the second photo task of the original queue is updated to become the head photo task, and only then will the new head photo task be sent to the IoT device side. In summary, by using this method, by completing the photo tasks one by one, the IoT device can correctly send photos back to the corresponding application when it receives photo requests sent by multiple applications on the electronic device, thereby improving the user experience.

[0026] In a third aspect, the present application also provides a collaborative work system, which includes one or more electronic devices provided in the second aspect, and one or more Internet of Things devices.

[0027] In a possible implementation, the collaborative working system may include two or more electronic devices, and each of the at least two electronic devices establishes a transmission channel with the Internet of Things device. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Indicate the scene Figure 1 ;

[0029] Figure 2a A schematic diagram of a center-side device provided in this application;

[0030] Figure 2b A schematic diagram of the software structure of the central-side device provided in this application;

[0031] Figure 3a A schematic diagram of an IoT device provided in this application;

[0032] Figure 3b A schematic diagram of the software structure of the IoT device provided for this application;

[0033] Figure 4 The process of the photographing method provided in the embodiment of the present application Figure 1 ;

[0034] Figure 5 This is an architectural diagram of a method for implementing a photographing method provided in an embodiment of the present application;

[0035] Figure 6a Flowchart 2 of the photographing method provided in an embodiment of the present application;

[0036] Figure 6b A flowchart of another photographing method provided in an embodiment of the present application;

[0037] Figure 7 This is an architectural diagram of another method for implementing a photographing method provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a collaborative working system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0041] The terms "first" and "second" and the like in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects.

[0042] In order to enable people skilled in the art to more clearly understand the solution of the present application, the application scenario of the technical solution of the present application is first described below.

[0043] Online education is gaining popularity, and students' demand for it is also growing. In some scenarios, when encountering unfamiliar words, students can look up the relevant explanations online. In other scenarios, online reading of textbook content facilitates students' learning and pronunciation. In still other scenarios, students need to submit their homework online. Therefore, how to meet users' online education needs based on smart devices is a challenge that needs to be addressed.

[0044] See also Figure 1 , the picture shows the scene Figure 1 .

[0045] The central-side device 10 is installed with an educational application. IoT devices 20 such as desk lamps take photos using their cameras and transmit the photos back to the central-side device to implement functions such as answering questions and submitting homework. Alternatively, the central-side device 10 uses the camera on the IoT device 20 to preview the images and provide users with various online education functions based on the images collected by the IoT device 20, such as online word lookup and online reading.

[0046] The central side device 10 may be a mobile phone, a tablet computer, a laptop computer, or a desktop computer, etc., and is not specifically limited in the embodiment of the present application.

[0047] The central device 10 and the IoT device 20 can perform near-field communication and / or far-field communication. Near-field communication can be accomplished through devices such as routers to exchange information between devices, while far-field communication can be accomplished through cloud servers to exchange information between devices.

[0048] The hardware architecture of the center-side device 10 will be described below.

[0049] See also Figure 2a , this figure is a schematic diagram of a center-side device provided in this application.

[0050] Figure 2a The center-side device 10 shown is merely an example, and the center-side device 10 may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 2a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0051] The central device 10 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0052] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU).

[0053] Among them, different processing units can be independent devices or integrated into one or more processors.

[0054] The controller can be the nerve center and command center of the central side device 10. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0055] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0056] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the central device 10, or to transfer data between the central device 10 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0057] The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger via the USB interface 130.

[0058] In some wireless charging embodiments, the charging management module 140 may receive wireless charging input via a wireless charging coil of the central device 10. While charging the battery 142, the charging management module 140 may also power the electronic device via the power management module 141.

[0059] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0060] The wireless communication function of the central side device 10 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0061] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the central device 10 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0062] For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

[0063] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G, etc. applied to the central side device 10. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0064] The wireless communication module 160 can provide wireless communication solutions applied to the central side device 10, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0065] In some embodiments, antenna 1 of the central device 10 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, so that the central device 10 can communicate with the network and other devices through wireless communication technology.

[0066] The central device 10 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0067] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the central device 10 may include one or N display screens 194, where N is a positive integer greater than one.

[0068] In the embodiment of the present application, the display screen 194 can display a shooting preview interface and a photo image interface, etc. It should be noted that in the embodiment of the present application, the shooting preview interface refers to an interface in which a user can view images captured in real time by the camera of the IoT device through the display screen 194.

[0069] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the central device 10. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage.

[0070] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the central-side device 10 by running the instructions stored in the internal memory 121, for example, enabling the central-side device 10 to implement the collaborative working method in the embodiment of the present application. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.

[0071] The data storage area can store data created during the use of the central device 10 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory and a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0072] The central device 10 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0073] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0074] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The central device 10 can listen to music or make hands-free calls through the speaker 170A. In some embodiments, the central device 10 can be equipped with multiple speakers 170A.

[0075] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the central device 10 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0076] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The central-side device 10 can be provided with at least one microphone 170C. In other embodiments, the central-side device 10 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the central-side device 10 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0077] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0078] Next, the software structure of the center-side device 10 will be described.

[0079] See also Figure 2b , this figure is a schematic diagram of the software structure of the central side device provided in this application.

[0080] The software system of the central side device 10 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the central side device 10.

[0081] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, these layers, from top to bottom, are the application layer, application framework layer, system library layer, extension layer, and kernel layer.

[0082] The application layer can include a series of application packages. For example, an application package can include applications such as gallery, map, Wi-Fi, Bluetooth, SMS, music, call, navigation, video, camera, device management application, and education application.

[0083] Among them, the device management application can sense, discover and register IoT devices in the area.

[0084] Educational applications can be developed by third-party manufacturers to provide users with online education functions, which may have but are not limited to viewing, broadcasting, annotation, query and other functions.

[0085] In some embodiments, a device management application can be bound to an IoT device such as a desk lamp; an educational application can also be bound to an IoT device such as a desk lamp.

[0086] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0087] like Figure 2b As shown, the application framework layer may include DVSDK, DMSDK, AUTH, intelligent interconnection services and interconnection services, etc.

[0088] Among them, identity verification (AUTH), also known as "authentication" and "authorization", refers to the use of certain means and interface calls for authentication to provide security permission management capabilities and complete the confirmation of user identity, that is, to provide authentication services.

[0089] DVSDK provides device virtualization services, also known as hardware virtualization services, though this distinction is not made in the following descriptions. It provides an entry point for third-party applications to connect to these services and manages interface access security permissions. Specifically, it establishes a logical channel between the central device and IoT devices, providing virtualized camera capabilities.

[0090] DMSDK is used to provide device management services, providing third-party manufacturers with far-field (i.e., cloud) IoT device information and near-field (i.e., nearby connectable IoT device information).

[0091] Smart interconnection services are used to provide physical transmission channels, data transmission capabilities, and manage the startup of interconnection services.

[0092] Interconnection services establish logical channels between central devices and IoT devices, providing virtualized camera capabilities. They are used to implement data processing, transmission channels, flow control, and capability collection.

[0093] In addition, it can also include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0094] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0095] Content providers store and retrieve data and make it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views.

[0096] For example, the display interface including the SMS notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.

[0097] The system layer includes system libraries and Android Runtime.

[0098] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0099] The core library consists of two parts: one containing the Java language's callable functions and the other the Android core library. The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0100] In the embodiment of the present application, Android Runtime also includes a virtual camera adaptation layer that provides the ability to register a virtual camera.

[0101] The system library can include multiple functional modules, such as multimedia platform, audio framework, graphics and image processing library, decoding module, and virtual camera adaptation layer.

[0102] The multimedia platform can be used for multimedia management, supporting playback and recording of a variety of common audio and video formats, as well as static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0103] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0104] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0105] The extension layer includes the Hardware Abstraction Layer (HAL). HAL is a routine package of the software layer and an interface layer located between the operating system kernel and the hardware circuit. It is used to simulate the details of a specific system platform so that programs can directly access hardware resources.

[0106] The HAL layer includes but is not limited to: Audio HAL, Sensor HAL, Modem HAL, Camera HAL, and Virtual Camera HAL.

[0107] The audio HAL is used to process audio streams, for example, by performing noise reduction and directional enhancement. The camera HAL is used to process image streams corresponding to the camera on the electronic device side, and the virtual camera HAL is used to process image streams corresponding to the virtual camera registered on the electronic device side, that is, to process image streams captured by the camera on the IoT device side.

[0108] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, camera drivers, USB drivers, CPU drivers, audio drivers, network drivers (such as Wi-Fi drivers), storage drivers, and printer drivers.

[0109] Next, the hardware structure of the IoT device 20 will be described.

[0110] See also Figure 3a , which is a schematic diagram of an IoT device provided by this application.

[0111] In some embodiments, the IoT device is a desk lamp with a camera. Figure 3a The IoT device 20 shown is merely an example and may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 3a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0112] The IoT device may include: a processor 210, a camera 201, a wireless communication module 202, a memory 203, an audio module 204, a USB interface 205, a charging management module 206, a power management module 207, a battery 208, a lighting device 209, a button 211, etc.

[0113] The processor 210 may include one or more processing units, for example, the processor 210 may include a GPU, an ISP, a controller, a memory, a video codec, etc. The different processing units may be independent devices or integrated into one or more processors.

[0114] The controller can be the nerve center and command center of the IoT device 20. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0115] Camera 201 is used to capture still images or videos. The object is projected through the lens into an optical image, which is then projected onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.

[0116] The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. Taking a desk lamp as an example, the camera 201 can be set on the desk lamp stand to capture images downward.

[0117] The IoT device 20 can implement a shooting function through an ISP, a camera 201, a video codec, a GPU, and the like.

[0118] The ISP processes data fed back by camera 201. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 201.

[0119] The wireless communication module 202 can provide wireless communication solutions for IoT device 20, including WLAN (such as Wi-Fi network) and Bluetooth (BT). In some embodiments, the antenna of IoT device 20 is coupled to the wireless communication module 202, so that IoT device 20 can communicate with the network and other devices through wireless communication technology.

[0120] The memory 203 can be used to store computer-executable program code, which includes instructions. The processor 210 executes the instructions stored in the memory 203 to execute various functional applications and data processing of the IoT device 20, for example, enabling the IoT device 20 to implement the collaborative working method in the embodiment of the present application.

[0121] The IoT device 20 can implement audio functions, such as music playback, through the audio module 204, the speaker 212, etc.

[0122] The USB interface 205 is an interface that complies with USB standards, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 205 can be used to connect a charger to charge the IoT device 20, and can also be used to transmit data between the IoT device 20 and peripheral devices.

[0123] The charging management module 206 is configured to receive charging input from a charger. While charging the battery 208 , the charging management module 206 can also power the IoT device 20 through the power management module 207 .

[0124] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives input from the battery 208 and / or the charging management module 206 to provide power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, and the lighting device 209.

[0125] The buttons 211 include a power button (or power button) and the like.

[0126] Next, the software structure of the IoT device 20 will be described.

[0127] See also Figure 3b , which is a schematic diagram of the software structure of the IoT device provided by this application.

[0128] The layers of an IoT device communicate with each other through software interfaces. In some embodiments, from top to bottom, they are the application layer, the application framework layer, the system library layer, and the kernel layer.

[0129] The application layer can include device application services. Device application services can be understood as system-level applications. Device application services are started after the IoT device system is started.

[0130] The application framework layer includes some predefined functions. Figure 3b As shown, the application framework layer may include intelligent interconnection services, resource managers and interconnection services, etc.

[0131] The resource manager is used to provide various resources, such as localized strings, icons, images, layout files, audio files, video files, etc.

[0132] Smart interconnection services provide physical transmission channels to provide data transmission capabilities and manage the activation of interconnection services.

[0133] Interconnection services: Establish a logical channel between central devices and IoT, provide virtualized camera capabilities, and provide open interfaces for device cameras. Specifically, these interfaces include data processing, capability acquisition, virtual audio modules, and virtual camera modules.

[0134] The system layer can include multiple functional modules, such as a multimedia platform, an audio framework, a graphics and image processing library, a decoding module, and an IoT vendor-adapted camera module.

[0135] The multimedia platform can be used for multimedia management, supporting playback and recording of a variety of common audio and video formats, as well as static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0136] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0137] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0138] IoT manufacturers adapt camera modules to implement interfaces for interconnected modules, providing functions such as opening the camera, taking photos, and previewing.

[0139] The kernel layer is the layer between hardware and software. The kernel layer may include camera drivers, USB drivers, CPU drivers, audio drivers, network drivers, storage drivers, etc.

[0140] Currently, by installing components such as DVSDK and DMSDP on central-side devices such as mobile phones and tablets, and setting up corresponding DMSDP on IoT devices such as desk lamps for adaptation, central-side devices can be connected to IoT devices and the virtualized camera function of IoT devices can be used on central-side devices.

[0141] At present, a typical application scenario of the above technology is the education scenario. The central-side device 10 is installed with an educational application, and the IoT device 20, such as a desk lamp, includes a camera. After the central-side device 10 establishes a connection with the IoT device 20, the virtualized camera function of the IoT device 20 can be used on the central-side device 10. For example, the camera of the IoT device 20 is used to capture the text of a book on the desktop. When the mobile phone is pointed at the text, the educational application on the central-side device 10 automatically recognizes the text, displays annotations, and performs voice broadcast. The educational application can also issue a photo request. After the IoT device, such as a desk lamp, takes a photo using the camera, the photo is transmitted back to the central-side device to implement functions such as answering questions and submitting homework.

[0142] However, in the current solution, once an IoT device establishes a transmission channel with a central device, it can only process a single photo request from a single application on the central device. If multiple applications on the central device send photo requests to the IoT device, photo transmission errors may occur, or even the virtual camera's photo service may be interrupted, impacting the user experience.

[0143] In order to solve the above technical problems, this application provides a photo-taking method, an electronic device and a collaborative working system, which enables the IoT device to correctly send photos back to the corresponding applications when it receives photo-taking requests sent by multiple applications on the electronic device, thereby improving the user experience.

[0144] The following describes the specific implementation method.

[0145] To help those skilled in the art better understand the technical solution of this application, the following first describes the process of establishing a connection between a central device and an IoT device and collaboratively taking photos in this application solution. The following description uses a tablet as the central device and a desk lamp as an example.

[0146] See also Figure 4 and Figure 5 .in, Figure 4 The process of the photographing method provided in the embodiment of the present application Figure 1 ; Figure 5 This is an architectural diagram of a method for implementing a photographing method provided in an embodiment of the present application.

[0147] in Figure 5 Only indicate Figure 4 The correspondence between the processes and modules in the middle part

[0148] 0. The device service initialization phase includes the following steps:

[0149] S0.1: In response to the user operation, the device application service of the desk lamp is started and the interconnection service is loaded.

[0150] For example, the user operation can be turning on the power of a desk lamp. In response to the user operation, the desk lamp system starts, the device application service starts, and the interconnection service is loaded. The interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp to provide data transmission capabilities.

[0151] S0.2, the lamp's Internet service loads the hardware abstraction service.

[0152] The interconnection service can also control the startup of the hardware abstraction service. For example, after the interconnection service is started, the interconnection service can load the hardware abstraction service as a plug-in. The hardware abstraction service can be used to establish a logical channel between the tablet and the desk lamp, providing virtualized camera capabilities and an open interface for the desk lamp camera.

[0153] The hardware abstraction service may include at least a basic component and a camera component. In the device service initialization phase, the interconnected service first loads the basic component and initializes the basic component. After the basic component is initialized, information interaction can be performed with the device adaptation module of the desk lamp to obtain device information and virtualization capability information. Exemplarily, the device information includes but is not limited to the device name, device identification, device type, etc. Exemplarily, the virtualization capability information includes but is not limited to whether a virtualized camera is supported, whether a virtualized microphone is supported, etc., and may further include supported virtualized camera functions, such as only supporting video preview, only supporting photo taking, or supporting both photo taking and video preview.

[0154] Among them, the desk lamp has the ability to support virtualized cameras, which can be understood as the camera of the desk lamp allowing other electronic devices (such as tablets) to call it, that is, it can be understood as the camera of the desk lamp allowing it to be used as a virtual camera of other electronic devices.

[0155] After the basic component obtains the device information and capability information of the desk lamp, if the desk lamp has the ability to support virtualized cameras, the basic component loads the camera component to provide the capability of virtualized cameras. At this point, the basic component can prepare to establish a negotiation channel, waiting to establish a negotiation channel with the tablet to negotiate network connection related information (including but not limited to IP addresses and ports, etc.). Among them, when the basic component prepares to establish a negotiation channel, it creates a session service (SessionServer) and sends the session name (Session Name) of the session service to the interconnection service, waiting for the transmission management service on the tablet side to establish a negotiation channel with the interconnection service on the desk lamp side.

[0156] 1. The device discovery phase includes the following steps:

[0157] S1.1: In response to a user operation, the tablet's education app sends a device discovery instruction to the device management service.

[0158] The user operation may be an operation in which the user clicks on a function option in the education APP that requires calling a virtual camera. For example, the user operation may be an operation in which the user clicks on a reading function, a word lookup function, an assignment function, or a photo taking function in the education APP.

[0159] The tablet's education app receives a user action and, in response, sends a device discovery command to the tablet's device management service. This device discovery command instructs the tablet to search for IoT devices that can connect to the tablet. For example, the device discovery command may include, but is not limited to, the command type and the type of device to be discovered. In this embodiment, the device discovery command specifically searches for a desk lamp that can connect to the tablet.

[0160] S1.2: The device management service in the tablet calls the authentication service to authenticate the education app and obtains the authentication result of the education app.

[0161] After receiving the device discovery instruction, the device management service authenticates the education APP based on the name of the education APP.

[0162] The authentication service obtains the authentication result of the education APP (authentication success or authentication failure) and sends the authentication result to the device management service.

[0163] S1.3: When the education APP is successfully authenticated, the device management service in the tablet sends a device search instruction to the transmission management service.

[0164] The device search instruction may include, but is not limited to, the instruction type, the device type to be searched, and the search method. Exemplary search methods include, but are not limited to, near-field device scanning and obtaining device information from a cloud server. In this embodiment, the device type to be searched is a desk lamp.

[0165] S1.4: The transmission management service in the tablet obtains a list of near- and far-field devices according to the device search instruction, and sends the list of near- and far-field devices to the device management service.

[0166] The far-field and near-field device lists include both the far-field device list and the near-field device list. The far-field devices listed in the far-field device list are registered devices obtained from the cloud server, while the near-field devices listed in the near-field device list are devices scanned via near-field communication. Device information in both the far-field and near-field device lists includes, but is not limited to, the device name, device ID, and device type.

[0167] When the transmission management service receives a device search instruction, it performs relevant device search operations based on the device type to be searched and the search method carried in the device search instruction, such as performing a near-field device scanning operation and obtaining relevant device information in the cloud server, obtaining a far-field device list and a near-field device list, and sending the far-field device list and the near-field device list to the device management service.

[0168] S1.5: The device management service in the tablet filters devices based on the far-field and near-field device lists, and reports the filtered device information to the education app.

[0169] Filter devices, determine the desk lamp information that can be linked with the tablet, and send it to the education APP.

[0170] The device management service can perform an intersection operation on the far-field device list and the near-field device list, filtering out lamps that only exist in the far-field device list or only exist in the near-field device list. The lamps that exist in both the far-field device list and the near-field device list are treated as lamps that can interact with the tablet. In this way, the device management service can filter out lamps that are not registered with the cloud server and can also filter out lamps that cannot perform near-field communication with the tablet.

[0171] In another optional embodiment, regardless of whether the tablet and the desk lamp are in the same local area network, the tablet's transmission management service can obtain a communication device list and a registered device list based on a device search instruction. The devices included in the communication device list are devices scanned via near-field communication or far-field communication, and the devices included in the registered device list are registered devices obtained from the cloud server. The near-field devices included in the near-field device list are devices scanned via near-field communication. In the communication device list and the registered device list, device information includes, but is not limited to, device name, device ID, device type, etc.

[0172] The transmission management service in the tablet sends the communication device list and the registered device list to the device management service. The device management service filters the devices based on the communication device list and the registered device list, and reports the filtered device information to the education APP. Among them, the device management service can perform an intersection operation on the communication device list and the registered device list, filtering out desk lamps that only exist in the communication device list or only exist in the registered device list, and using desk lamps that exist in both the communication device list and the registered device list as desk lamp information that can be linked with the tablet. In this way, the device management service can filter out desk lamps that are not registered in the cloud server, and can also filter out desk lamps that cannot perform near-field communication with the tablet.

[0173] 2. The virtual camera enabling stage includes the following steps:

[0174] S2.1: The tablet's education APP determines the desk lamp to be linked.

[0175] S2.2: The tablet's educational app performs device verification and device connection on the desk lamp, and obtains the desk lamp's ability to support a virtualized camera.

[0176] S2.3: The tablet's education app sends a virtual camera enablement request to the hardware virtualization service.

[0177] The virtual camera enable request is used to instruct the virtual camera to be registered in the virtual camera HAL. The virtual camera enable request may include, but is not limited to, a request type, a device name, a device identifier, a device type, and an identifier of the virtual camera.

[0178] S2.4: The hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

[0179] After receiving the virtual camera enable request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enable request.

[0180] S2.5: After the virtual camera registration is completed, the tablet's virtual camera HAL sends a virtual camera enabling success indication to the education app.

[0181] 3. Virtual camera preview access stage:

[0182] S3.1: The hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

[0183] The virtual camera access instruction refers to an instruction for calling a virtual camera, wherein the virtual camera access instruction may include but is not limited to an instruction type, a virtual camera ID, and camera configuration parameters.

[0184] S3.2: The camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

[0185] After receiving the virtual camera access instruction, the camera service generates a corresponding image preview request based on the virtual camera ID and sends it to the virtual camera HAL.

[0186] The image preview request is used to request a preview of an image data stream. Exemplarily, the image preview request may include, but is not limited to, a request identifier, a virtual camera ID, and camera configuration parameters.

[0187] S3.3: The virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0188] After receiving the image preview request, the virtual camera HAL determines the matching virtualized hardware identifier based on the virtual camera ID carried in the image request.

[0189] In this embodiment, the virtual camera HAL determines the linked desk lamp camera based on the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request based on the determined virtualization hardware identifier and sends it to the hardware virtualization service.

[0190] Exemplarily, the image preview request may include but is not limited to a request identifier, device information (ie, desk lamp information), a virtualized hardware identifier (ie, desk lamp camera identifier), and camera configuration parameters.

[0191] S3.4: The hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0192] The hardware virtualization service sends an image preview request to the transmission management service. The image preview request may include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters.

[0193] When the tablet's hardware virtualization service sends an image preview request to the transmission management service, if it finds that a data channel with the desk lamp is not established, it generates a data channel establishment request and sends it to the transmission management service. This data channel establishment request is used to instruct data transmission between the desk lamp and the tablet. The data channel establishment request may include, but is not limited to, a session identifier, connection information, and data encoding and decoding mode.

[0194] The tablet's transmission management service receives the data channel establishment request and, based on the information in the data channel establishment request, establishes a data channel connection with the lamp, effectively establishing a data channel between the tablet and the lamp. The tablet's transmission management service and the lamp's interconnection service can then use this data channel to transmit various data, including but not limited to image data.

[0195] After the data channel is successfully established, the tablet's transport management service sends a successful data channel connection indication to the tablet's hardware virtualization service, and the lamp's interconnection service sends a successful data channel connection indication to the camera component in the hardware abstraction service. The successful data channel connection indication may include, but is not limited to, a successful connection flag and data channel-related information.

[0196] S3.5: The transfer management service in the tablet transmits an image preview request to the interconnection service in the desk lamp.

[0197] The transmission management service in the tablet determines the corresponding control channel based on the device information carried in the image preview request, and transmits the image preview request to the interconnection service of the desk lamp through the control channel.

[0198] S3.6: The interconnection service in the desk lamp sends an image preview request to the camera driver.

[0199] After receiving the image preview request, the interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding image preview request to the camera driver.

[0200] S3.7: The camera driver in the desk lamp drives the camera to capture images and transmits the preview image data to the hardware virtualization service of the tablet through the data channel.

[0201] The camera driver turns on the camera and drives it to capture images using the camera configuration parameters carried in the image preview request. This generates a preview image data stream and sends it to the interconnection service via the hardware abstraction service. The interconnection service then continuously transmits the preview image data stream to the tablet's hardware virtualization service via the data channel. The details of the preview image data stream's packetization, assembly, and encoding / decoding are not detailed here.

[0202] S3.8: The hardware virtualization service in the tablet sends the preview image data to the virtual camera HAL.

[0203] The hardware virtualization service continuously receives the preview image data stream and sends the preview image data stream to the virtual camera HAL.

[0204] S3.9: The virtual camera HAL in the tablet sends the preview image data to the camera service.

[0205] At this time, the virtual camera HAL continues to obtain the preview image data collected by the desk lamp camera and continues to send the preview image data to the camera service.

[0206] S3.10: The camera service in the tablet sends the preview image data to the education app.

[0207] S3.11: The educational app on the tablet displays a preview image.

[0208] 4. Virtual camera photo taking stage

[0209] S4.1: In response to the received user operation, the education APP in the tablet sends a photo request to the hardware virtualization service.

[0210] The user operation may be, for example, clicking a photo option. In response to the received user operation, the education APP in the tablet sends a photo request to the hardware virtualization service.

[0211] The photo request may include but is not limited to the image sequence number, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters, etc. The camera configuration parameters include but are not limited to image resolution.

[0212] The photo request may also carry a task identifier to ensure orderly management of multiple photo tasks.

[0213] S4.2: The hardware virtualization service in the tablet sends a photo-taking request to the transmission management service.

[0214] S4.3: The transfer management service in the tablet transmits a photo-taking request to the interconnection service of the desk lamp.

[0215] The transmission management service in the tablet determines the corresponding control channel based on the device information carried in the image preview request, and transmits the image preview request to the interconnection service of the desk lamp through the control channel.

[0216] S4.4: The Internet service in the desk lamp sends a photo request to the camera driver.

[0217] After receiving the image preview request, the interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding photo request to the camera driver.

[0218] S4.5: The camera driver in the desk lamp drives the camera to capture images and transmits the captured image data to the hardware virtualization service of the tablet through the data channel.

[0219] The camera driver drives the camera to capture an image using the camera configuration parameters carried in the photo request, obtains the captured image data, and sends the captured image data to the interconnection service through the hardware abstraction service. The interconnection service then continuously transmits the captured image data to the tablet's hardware virtualization service through the data channel. The image data packetization, grouping, encoding, and decoding processes are not detailed here.

[0220] S4.6: The hardware virtualization service in the tablet sends the captured image data to the education app.

[0221] S4.7: The educational app on the tablet displays the captured image.

[0222] In the above process, the hardware abstraction service on the IoT device side is also the hardware virtualization service in this application solution.

[0223] This technical solution allows central devices to be directly connected to electronic devices such as mobile phones and tablets. IoT devices can be installed separately from central devices, eliminating the need for a heavy base with a retractable camera. IoT devices, such as desk lamps, can be positioned freely, with screen angles adjusted as needed. Furthermore, educational applications installed on central devices offer high compatibility and scalability, making them easy to implement and promote, greatly facilitating education.

[0224] The above solution describes how a central device uses a virtual camera service to take photos with an IoT device. It also illustrates the process of a single app requesting a photo. The following example uses a central device running two apps and the IoT device receiving photo requests from both apps. It's understood that the principles are similar when an IoT device receives photo requests from three or more apps, so we won't elaborate further here.

[0225] The following first explains the meaning of the nouns involved in the method.

[0226] The device ID is used to identify individual cameras. An IoT device can include one or more cameras, each with a different device ID. For example, camera 1 has a device ID of 1, and camera 2 has a device ID of 2. If an IoT device has only one camera, the device ID is fixed, for example, 2. Central devices also have a device ID, for example, 1.

[0227] The service ID refers to the virtualized service that the IoT device can provide. For example, a service ID of 1 represents a virtualized camera service, and a service ID of 2 represents a virtualized microphone service.

[0228] The following method uses a tablet computer (referred to as tablet) as the central device and an electronic desk lamp as the IoT device as an example.

[0229] See also Figure 6a , this figure is the second flowchart of the photo-taking method provided in an embodiment of the present application.

[0230] The method comprises the following steps:

[0231] S601: Receive i photo-taking requests in sequence.

[0232] i photo requests are issued by j applications, each application issues at least one photo request, i and j are integers greater than or equal to 1, and i is greater than or equal to j.

[0233] S602: inserting the photo request into the photo request queue in sequence according to the order in which the i photo requests are received, and the first photo request in the photo request queue is the first photo request received.

[0234] S603: Send the nth photo request to the IoT device to obtain photo data. When the photo task of the nth photo request is completed, delete the nth photo request from the photo request queue until there is no photo request in the photo request queue, and n is 1, 2, ..., i in sequence.

[0235] The solution provided by the embodiment of the present application utilizes a photo request queue to arrange various photo tasks, and the photo request queue is executed in the order of first-in-first-out and last-in-last-out. The photo tasks in the photo request queue are completed in sequence to avoid sending multiple tasks to the IoT device side at the same time, which may cause a crash on the IoT device side or an error in the photo data being sent back. Completing photo tasks one by one means: sending the photo request at the head of the photo request queue to the IoT device, and after receiving the photo data sent back by the IoT device, sending the photo data back to the correct corresponding application. At this time, it is considered that the photo task at the head of the photo request queue is completed. Then the head photo task of the photo request queue is deleted. At this time, the second photo task of the original queue is updated to become the head photo task, and only then will a new head photo task be sent to the IoT device side. In summary, by using this method, by completing the photo tasks one by one, the IoT device can correctly send photos back to the corresponding application when it receives photo requests sent by multiple applications on the electronic device, thereby improving the user experience.

[0236] The following describes the specific implementation method.

[0237] See also Figure 6b and Figure 7 .in, Figure 6b A flowchart of another photographing method provided in an embodiment of the present application; Figure 7 This is an architectural diagram of a method for implementing a photographing method provided in an embodiment of the present application.

[0238] The specific implementation method of establishing a connection and transmission channel between the IoT device and the central side device can be found in the description of the above embodiments, and the embodiments of this application will not be repeated here.

[0239] The method comprises the following steps:

[0240] S5.1: The remote device management object receives the photo taking request 10001 from APP1.

[0241] In the embodiment of the present application, each APP corresponds to an APP callback function, or simply called back (Callback, i.e., call then back, abbreviated as cb), which is used to return to the main function after being called by the main function for calculation.

[0242] The callback function of APP1 is cb1, the callback function of APP2 is cb2, and so on. There is a mapping relationship between the callback function and the APP.

[0243] Each photo request sent by an APP corresponds to a request code.

[0244] In some embodiments, for an APP, the request code starts from a first preset value. Each time the APP issues a photo request, the corresponding request code increases by 1 until it reaches a second preset value and then stops, and then the cycle starts again from the first preset value. The first preset value and the second preset value are not specifically limited in this embodiment of the application. Taking the first preset value as 10001 and the second preset value as 20000 as an example, the request code of the first photo request issued by the APP is 10001, the request code of the second photo request is 10002, and so on. When the number reaches 20000, the cycle count starts again from 10001.

[0245] The request codes of the photo request sent by each APP are counted independently and do not affect each other. The first preset value and the second preset value corresponding to each APP can be the same or different, and this embodiment of the application does not specifically limit this.

[0246] S5.2: The remote device object receives the photo taking request 10001 from APP1.

[0247] The remote device object determines, based on APP1's photo request 10001, that the object to be controlled is the remote IoT device, the desk lamp Device2.

[0248] S5.3: The remote service receives the photo taking request 10001 from APP1.

[0249] The remote service determines that the requested service ID (service ID) is service 1 based on the photo request 10001 of APP1, that is, the requested service is the virtualized camera service.

[0250] S5.4: The remote service checks whether the current camera logical channel exists.

[0251] The camera logical channel is the data transmission channel established between the central device and the IoT device.

[0252] The collaborative working system of this application supports the release of idle channels. When the camera logical channel established between the central device and the IoT device has not transmitted data for a long time, the channel will be destroyed to free up space and reduce resource usage. Therefore, when a previously established camera logical channel is destroyed due to a long period of inactivity, a new channel needs to be established.

[0253] Therefore, it is necessary to determine whether the current camera logical channel exists. If so, there is no need to re-establish it and the camera command can be directly transmitted through the channel. If not, execute S5.5.

[0254] S5.5: The remote service determines that the current camera logical channel does not exist.

[0255] S5.6: Create a camera logical channel and cache APP1's photo request 10001.

[0256] For the specific implementation of creating camera logical channels, please refer to the above Figure 4 Step S3.4 in the relevant description, that is, the process of establishing a data channel, will not be repeated here.

[0257] In this embodiment, a photo management module is added to the hardware virtualization service of the central device. This module is used to manage photo tasks. Specifically, each photo request is placed into a photo request queue, achieving a unique photo sequence. The photo request queue is characterized by first-in, first-out, meaning that earlier photo requests are processed first, followed by later ones.

[0258] The embodiment of the present application does not specifically limit the maximum number of photo-taking requests pre-allocated to the queue that is allowed to queue.

[0259] The contents of the cache at this time can be seen in Table 1. The request queue number and delay in Table 1 are only schematic illustrations and do not constitute a limitation on the technical solution of the present application. In addition, in other embodiments, the time consumption can also be replaced by a specific timestamp, or the time consumption and the timestamp can be cached at the same time. The timestamp can record the issuance time and / or cache time of the corresponding photo request, which is not limited here. Adding information such as the time consumption or timestamp can avoid the overlap of the newly cached content and the cached content corresponding to the old channel after the camera logical channel is destroyed and rebuilt.

[0260] Table 1: Cache contents Table 1

[0261] Request queue number state time consuming Request Encoding 00001 Waiting to send 0ms 10001

[0262] In some embodiments, the request queue number starts at a third preset value. Each time a photo request is queued in the request queue, the corresponding request queue number increments by 1 until it reaches a fourth preset value, then stops incrementing and cyclically increments again, starting from the third preset value. The third and fourth preset values ​​are not specifically limited in this embodiment of the application. For example, if the third preset value is 00001 and the fourth preset value is 10000, the first photo request in the request queue will have a request queue number of 00001, the second photo request will have a request queue number of 00002, and so on. When the number reaches 10000, the request queue number will restart from 00001. Unlike independent request codes for each app, photo requests issued by each app are recorded using the same request queue number. That is, if photo request 10001 issued by APP 1 is the first in the queue, the request queue number is 00001; if photo request 10001 issued by APP 2 is the second in the queue, the request queue number is 00002.

[0263] S5.7: Return a message indicating the camera logical channel is successfully created to the remote service.

[0264] S5.8: The remote service reads the cache status.

[0265] The remote service reads the photo taking request 10001 sent by APP1 which is at the top of the reading material photo taking request queue.

[0266] In some embodiments, the cache status of camera data processing may be updated after the remote service reads the cache status to refresh the time consumed in the cache.

[0267] The above describes the process of establishing a logical channel between the cache and the camera when the first photo request arrives. The following describes the process when other subsequent photo requests arrive.

[0268] 5.9: The remote service sends a photo request 10001 from App 1 to the IoT device.

[0269] This enables IoT devices to take photos.

[0270] S6.1: The remote device management object receives the photo taking request 10001 from APP2.

[0271] The callback function of APP2 is cb2.

[0272] S6.2: The remote device object receives the photo taking request 10001 from APP2.

[0273] The remote device object determines, based on APP2's photo request 10001, that the object to be controlled is the remote IoT device, the desk lamp Device2.

[0274] S6.3: The remote service receives the photo taking request 10001 from APP2.

[0275] The remote service determines that the requested service ID (service ID) is service 1 based on the photo request 10001 of APP2, that is, the requested service is the virtualized camera service.

[0276] That is, at this time, the photo request 10001 of APP2 can be arranged in a photo request queue with the photo request 10001 of APP1.

[0277] S6.4: Send a cache request to cache APP2's photo request 10001.

[0278] S6.5: Update cache status.

[0279] That is, the camera data processing updates the cache status. The updated cache status is shown in Table 2.

[0280] Table 2: Cache contents table 2

[0281] Request queue number state time consuming Request Encoding 00001 Waiting to return 500ms 10001 00002 Waiting to send 0ms 10001

[0282] S6.6: Return the cache to the remote service.

[0283] At this time, according to the status shown in Table 2, no photo data is obtained. At this time, continue waiting.

[0284] S7.1: The remote device management object receives the photo taking request 10002 from APP1.

[0285] APP1 sends another photo request. This time, the request code of the photo request increases by 1.

[0286] S7.2: The remote device object receives the photo taking request 10002 from APP1.

[0287] The remote device object determines, based on APP1's photo request 10002, that the object to be controlled is the remote IoT device, the desk lamp Device2.

[0288] S7.3: The remote service receives the photo taking request 10002 from APP1.

[0289] The remote service determines that the requested service ID (service ID) is service 1 based on the photo request 10002 of APP1, that is, the requested service is the virtualized camera service.

[0290] S7.4: Send a cache request to cache APP1's photo request 10002.

[0291] S7.5: Update cache status.

[0292] That is, the camera data processing updates the cache status. The updated cache status is shown in Table 3.

[0293] Table 3: Cache contents

[0294] Request queue number state time consuming Request Encoding 00001 Waiting to return 550ms 10001 00002 Waiting to send 50ms 10001 00003 Waiting to send 0ms 10002

[0295] S7.6: The IoT device takes a photo to obtain photo data.

[0296] S7.7: The IoT device returns image data to the central device through the camera logical channel.

[0297] S7.8: Return the image data to the camera data of the central device.

[0298] At this point, the image data corresponding to the photo request 10001 of APP1 has been obtained. The following describes the ending process of the photo request 10001 of APP1.

[0299] S8.1: Return the image data corresponding to APP1's photo request 10001 to the remote service and update the cache.

[0300] The updated cache status is shown in Table 4.

[0301] Table 4: Cache contents

[0302] Request queue number state time consuming Request Encoding 00002 Waiting to send 350ms 10001 00003 Waiting to send 300ms 10002

[0303] That is, the cached content of the image data corresponding to the photo request 10001 of APP1 can be deleted from the request queue at this time, thereby realizing the first-in-first-out of photo requests.

[0304] S8.2: The remote service returns the image data corresponding to APP1's photo request 10001 to the remote device object.

[0305] S8.3: The remote device object returns the image data corresponding to APP1's photo request 10001 to the remote device management object.

[0306] S8.4: The remote device management object returns the image data corresponding to APP1's photo request 10001 to the application.

[0307] At this point, the photo-taking task required by APP1's photo-taking request 10001 has been completed.

[0308] S8.5: The remote service sends a photo-taking request 10001 from APP2 to the IoT device.

[0309] S8.6: The IoT device takes a photo to obtain photo data.

[0310] The subsequent completion process of the photo-taking task is similar to the above steps and follows the first-in-first-out rule of the photo-taking request queue, so I will not go into details here.

[0311] It is understandable that the division of the above steps is only for the convenience of explanation and does not constitute a limitation on the technical solution of the present application. In actual applications, the order of the above steps can be appropriately adjusted. For example, the step of "the remote service sends the photo request 10001 issued by APP2 to the IoT device side" in S8.5 can be performed in advance, for example, after S8.1. For another example, the step of establishing a logical transmission channel starts when the first photo request in the photo request queue is received, but the time for the channel to be established may be after receiving multiple photo requests, that is, the time for establishing the channel is not limited.

[0312] Furthermore, since the logical transmission channel can be dynamically created or destroyed, the photo request queue adopted in the present application can also be dynamically created and destroyed, that is, when a photo request is received and the logical transmission channel is created, a photo request queue is generated; when the logical transmission channel is destroyed because no data transmission has been performed for a long time, there is no photo task at this time, and the photo request queue can be deleted to free up space and reduce resource usage.

[0313] In summary, the solution provided by the embodiments of the present application enables the IoT device to correctly transmit photos back to the corresponding applications when it receives photo-taking requests from multiple applications on the electronic device, thereby improving the user experience.

[0314] Based on the photographing method provided in the above embodiments, the present application also provides an electronic device.

[0315] Continue to see Figure 2a Schematic diagram of the electronic device shown.

[0316] The electronic device provided in the embodiment of the present application includes: a memory, namely an internal memory 121 and a processor 110.

[0317] The internal memory 121 is coupled to the processor 110 .

[0318] The internal memory 121 stores program instructions. When the program instructions are executed by the processor 110, the electronic device executes the photographing method described in the above embodiment.

[0319] Specifically, the electronic device receives i photo requests in sequence, where the i photo requests are issued by j applications, each application issues at least one photo request, i and j are integers greater than or equal to 1, and i is greater than or equal to j.

[0320] When a transmission channel for the photo data is not established between the electronic device and the IoT device, the electronic device establishes the transmission channel and creates the photo queue.

[0321] When the transmission channel has been established between the electronic device and the IoT device, the electronic device creates the photo queue. The electronic device sequentially inserts the i photo requests into the photo request queue in the order in which the i photo requests were received, and stores a correspondence between a photo request code, time information, and a request queue number. The photo request code is the code of the photo request issued by the application itself, and the request queue number is the sequence number of each photo request in the photo request queue.

[0322] In some embodiments, the photographing request code counts cyclically between a first preset value and a second preset value.

[0323] In some embodiments, the request queue number cyclically counts between a third preset value and a fourth preset value.

[0324] In some embodiments, the time information includes at least one of the time taken for the photo-taking task or the time when the photo-taking request was received.

[0325] The electronic device sends an nth photo-taking request to the IoT device through the transmission channel to obtain photo-taking data;

[0326] The electronic device returns the photographing data to the application sending the nth photographing request according to the correspondence between the application's callback function and the photographing request code, so as to complete the photographing task of the nth photographing request;

[0327] The electronic device deletes the correspondence between the photo request code of the nth photo request, the time information of the nth photo request, and the request queue number of the nth photo request from the photo request queue until there are no more photo requests in the photo request queue, and n is sequentially 1, 2, ..., i.

[0328] Furthermore, an embodiment of the present application also provides a collaborative work system.

[0329] See also Figure 8 , which is a schematic diagram of a collaborative work system provided in an embodiment of the present application.

[0330] The collaborative working system 30 includes: an electronic device 10 and an IoT device 20 .

[0331] The electronic device 10 is also the central side device. For the specific implementation of the electronic device 10 and the IoT device 20, please refer to the relevant description in the above embodiments, and the embodiments of this application will not be repeated here.

[0332] Multiple applications can be run simultaneously on the electronic device 10. After using the solution provided by the embodiment of the present application, when the IoT device receives photo-taking requests sent by multiple applications on the electronic device, it can correctly send photos back to the corresponding applications, thereby improving the user experience.

[0333] The electronic device may be a tablet computer, a mobile phone, a laptop computer, or a desktop computer, which is not specifically limited in the present embodiment. The IoT device may be a desk lamp.

[0334] This embodiment also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an IoT device, the IoT device executes the above-mentioned related method steps to implement the photo-taking method in the above-mentioned embodiment.

[0335] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the photographing method in the above-mentioned embodiment.

[0336] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the photographing method in the above-mentioned method embodiments.

[0337] Among them, the IOT device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0338] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0339] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A photographing method, characterized in that: Applied to an electronic device, the electronic device is used to take a photo through a camera of an Internet of Things device to obtain photo data, the method comprising: receiving i photo requests in sequence, the i photo requests being issued by j applications, each application issuing at least one photo request, wherein i and j are integers greater than or equal to 1, and i is greater than or equal to j; Inserting the i photo requests into a photo request queue in sequence according to the order in which the i photo requests are received, wherein the first photo request in the photo request queue is the first photo request received; Send the nth photo request to the IoT device to obtain photo data. When the photo task of the nth photo request is completed, delete the nth photo request from the photo request queue until there is no photo request in the photo request queue, where n is 1, 2, ..., i in sequence.

2. The photographing method according to claim 1, wherein: The step of sequentially inserting the i photo request queues according to the order in which the i photo request are received specifically includes: Creating the photo request queue; The photo requests are inserted into the photo request queue in sequence according to the order in which the i photo requests are received, and the correspondence between the photo request code, time information and request queue number is saved. The photo request code is the code of the photo request issued by the application itself, and the request queue number is the sequence number of each photo request in the photo request queue.

3. The photographing method according to claim 2, wherein: The step of creating the photo request queue specifically includes: When a transmission channel for the photo data is not established between the electronic device and the IoT device, establishing the transmission channel and creating the photo request queue; When the transmission channel has been established between the electronic device and the Internet of Things device, the photo request queue is created.

4. The photographing method according to claim 2, wherein: The photographing request code is cyclically counted between a first preset value and a second preset value.

5. The photographing method according to claim 2, wherein: The request queue number is cyclically counted between a third preset value and a fourth preset value.

6. The photographing method according to claim 2, wherein: The time information includes: At least one of the time it takes to complete the photo-taking task or the time it takes to receive the photo-taking request.

7. The photographing method according to claim 3, wherein: The method further comprises: When the transmission channel between the electronic device and the Internet of Things device is destroyed, the photo request queue is deleted.

8. The photographing method according to claim 3, wherein: Sending an nth photo request to the IoT device to obtain photo data, and when the photo task of the nth photo request is completed, deleting the nth photo request from the photo request queue, specifically comprising: Sending an nth photo-taking request to the IoT device through the transmission channel to obtain photo-taking data; Returning the photographing data to the application sending the nth photographing request according to the correspondence between the application's callback function and the photographing request code, so as to complete the photographing task of the nth photographing request; The correspondence between the photographing request code of the nth photographing request, the time information of the nth photographing request and the request queue number of the nth photographing request is deleted from the photographing request queue.

9. An electronic device, characterized in that: The electronic device comprises: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the photographing method according to any one of claims 1 to 8.

10. A collaborative work system, characterized in that: The system includes the electronic device described in claim 9 and also includes an Internet of Things device.

Images