Device control methods, apparatus, equipment, and storage media based on electronic manuals

Abstract

This application discloses a device control method, apparatus, electronic device, and readable storage medium based on an electronic manual, applicable to the field of smart device technology. The device control method based on the electronic manual includes: determining a corresponding target electronic manual based on the device information of the target device; matching interface description information corresponding to the current operating interface in the target electronic manual through image recognition of the current operating interface of the target device; and displaying the interface operation content corresponding to the interface description information based on a preset display interface. This application solves the technical problem of low efficiency in pairing electronic manuals and smart device operating interfaces.

Description

Technical Field

[0001] This application relates to the field of smart device technology, and in particular to a device control method, apparatus, electronic device, and readable storage medium based on an electronic instruction manual. Background Technology

[0002] With the continuous development of technology, smart devices are taking on more and more functions to better meet user experience needs. At the same time, the user interface of these devices is becoming increasingly complex. To overcome the shortcomings of paper manuals, such as easy loss and lack of timely updates, electronic manuals are typically used to describe functions and usage instructions, thus guiding users to correctly use the smart device's interface. Currently, users usually use their mobile phones' electronic manual function to find the corresponding electronic manual for their smart device, and then further search for the interface descriptions such as functions and icons within the electronic manual. In other words, the matching process between the current user interface and the interface descriptions in the electronic manual relies on manual identification.

[0003] However, because electronic manuals contain a lot of interface description information, users often spend a lot of time looking for the corresponding interface operation content in the electronic manual. Therefore, the current efficiency of pairing electronic manuals and smart device operation interfaces is low. Summary of the Invention

[0004] The main objective of this application is to provide a device control method, apparatus, electronic device, and readable storage medium based on an electronic instruction manual, aiming to solve the technical problem of low efficiency in pairing electronic instruction manuals and smart device operation interfaces in the prior art.

[0005] To achieve the above objectives, this application provides a device control method based on an electronic instruction manual, the device control method based on an electronic instruction manual comprising:

[0006] Based on the equipment information of the target device, determine the corresponding target electronic instruction manual;

[0007] By performing image recognition on the current operating interface of the target device, the interface description information corresponding to the current operating interface is matched in the target electronic instruction manual;

[0008] The interface operation content corresponding to the interface description information is displayed based on the preset display interface.

[0009] To achieve the above objectives, this application also provides a device control device based on an electronic instruction manual, the device control device based on an electronic instruction manual comprising:

[0010] The determination module is used to determine the corresponding target electronic instruction manual based on the device information of the target device;

[0011] The matching module is used to perform image recognition on the current operating interface of the target device and match the interface description information corresponding to the current operating interface in the target electronic manual.

[0012] The control module is used to display the interface operation content corresponding to the interface description information based on the preset display interface.

[0013] This application also provides an electronic device, the electronic device including: a memory, a processor, and a program of the device control method based on the electronic manual stored in the memory and executable on the processor, wherein when the program of the device control method based on the electronic manual is executed by the processor, it can implement the steps of the device control method based on the electronic manual as described above.

[0014] This application also provides a computer-readable storage medium storing a program implementing a device control method based on an electronic instruction manual. When the program is executed by a processor, it implements the steps of the device control method based on the electronic instruction manual as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the device control method based on the electronic instruction manual described above.

[0016] This application provides a device control method, apparatus, electronic device, and readable storage medium based on an electronic manual. Specifically, by determining the corresponding target electronic manual based on the device information of the target device, the purpose of matching the target device with the corresponding target electronic manual is achieved. Image recognition is performed on the current operating interface of the target device to match the interface description information corresponding to the current operating interface in the target electronic manual. Then, the interface operation content corresponding to the interface description information is displayed on a preset display interface. Since the target electronic manual corresponding to the target device is obtained, image recognition of the current operating interface of the target device automatically matches the interface description information of the current operating interface, and then displays the interface operation content corresponding to the interface description information on the preset display interface. Because the preset display interface can intuitively reflect the interface description information of the current operating interface, the user can directly obtain the interface operation content of the current operating interface on the preset display interface. That is, the purpose of automatically matching the current operating interface of the target device with the interface description information in the electronic manual is achieved, rather than requiring the user to perform a pairing process manually. Therefore, this overcomes the technical defect that users spend a lot of time searching for the corresponding interface operation content in the electronic manual, thus improving the efficiency of controlling the pairing of the electronic manual and the smart device's operating interface. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the device control method based on an electronic instruction manual according to this application;

[0020] Figure 2 This is a flowchart illustrating the second embodiment of the device control method based on an electronic instruction manual according to this application;

[0021] Figure 3 This application presents a scenario diagram illustrating the interface operation and control of a home IoT device based on an electronic instruction manual.

[0022] Figure 4 This is a schematic diagram of an embodiment of the device control device based on the electronic instruction manual of this application;

[0023] Figure 5 This is a schematic diagram of the hardware operating environment involved in the device control method based on the electronic instruction manual in the embodiments of this application.

[0024] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1

[0027] First, it should be understood that electronic manuals, with their advantages of being less prone to loss and easy to iterate, have gradually replaced traditional paper manuals. Currently, as smart devices become increasingly feature-rich, the corresponding electronic manuals also contain more and more information. This is especially true for smart devices with user interfaces. Matching the interface descriptions within the electronic manual to the corresponding information in the user interface can be very time-consuming. For example, consider a washing machine with a "schedule" button on its interface. While the word "schedule" might indicate the button's function, it doesn't necessarily mean the user fully understands how to use it. Currently, users typically find the washing machine's electronic manual on their smart device and then locate the "schedule" button's instructions within that manual. However, because the electronic manual contains information about buttons, icons, and text on the user interface, users spend considerable time searching on their smart device. Therefore, existing technologies for pairing electronic manuals and smart device user interfaces are inefficient. Thus, a more efficient method for pairing electronic manuals and smart device user interfaces is urgently needed.

[0028] This application provides a device control method based on an electronic manual. In the first embodiment of this application's device control method based on an electronic manual, refer to... Figure 1 The device control method based on the electronic instruction manual includes:

[0029] Step S10: Determine the corresponding target electronic instruction manual based on the device information of the target device;

[0030] Step S20: By performing image recognition on the current operating interface of the target device, the interface description information corresponding to the current operating interface is matched in the target electronic manual.

[0031] In this embodiment, it should be noted that the device control method based on the electronic instruction manual is set on an intelligent display device with a preset display interface and image recognition function, specifically a mobile phone, computer, or personal PC, etc. The device information is information that identifies the specific identity of the target device. The target device is a device with an operating interface. The device specifically includes machine equipment and intelligent devices, etc. The target device can be an air conditioner, television, washing machine, or car, etc., used to distinguish it from other devices in the same application scenario as the target device. For example, in one implementable method, assuming there is a computer, a television from manufacturer A, a television from manufacturer B, a washing machine, and a mobile phone in a certain application scenario, and the user wants to use the mobile phone to find the electronic instruction manual corresponding to the television from manufacturer A, then the device identification information is used to identify the target device as a television and the target device is produced by manufacturer A.

[0032] Additionally, it should be noted that the target electronic instruction manual is used to describe the target device, and the current operating interface is the operating interface of the target device in different states, used to display the operating content of the target device. Specifically, the current operating interface can be an operating interface in the factory state, an operating interface in the working state, or an operating interface in the non-working state. The interface description information is used to describe the current operating interface, and the interface description information can specifically be text information, icon information, and indicator information, etc. For example, in one implementable method, the current operating interface has icon C, text D, and indicator light E, then the interface description information includes a description of the meaning of icon C, the meaning of text D, and the meaning of indicator light E.

[0033] As an example, steps S10 to S20 include: obtaining device information input by the user; using the device information as an index to query the target electronic manual corresponding to the target device in a preset electronic manual table, wherein the preset electronic manual table is used to store the mapping relationship between the target device and the target device; obtaining the current operation interface input by the user; and matching the interface description information corresponding to the current operation interface in the target electronic manual by performing image recognition on the current operation interface.

[0034] In one feasible approach, the smart display device is a mobile phone. The user uses the "electronic manual" function on the phone to query the target electronic manual corresponding to the target device. If the user is using the "electronic manual" function for the first time, user registration and login are required. If the user is already using the "electronic manual" function, the user inputs device information via the phone, such as selecting the type and model of the target device on the device query page, to determine the target electronic manual corresponding to the target device. The phone then captures the current operating interface using its camera, allowing the phone's image recognition module to perform image recognition and match the interface description information corresponding to the current operating interface in the target electronic manual. Furthermore, the device information and the current operating interface can be input simultaneously. For example, when using the "electronic manual" function, only the user-input current operating interface can be captured. The phone then automatically retrieves device information on the current operating interface used to identify the target device, such as nameplates, product logos, or product names, thereby reducing human-computer interaction steps and further accelerating the one-to-one pairing process between the electronic manual and the device.

[0035] The interface description information includes identification description information and video description information. The step of performing image recognition on the current operating interface of the target device and matching the interface description information corresponding to the current operating interface in the target electronic manual includes:

[0036] Step A10: Obtain the content to be explained corresponding to the current operation interface;

[0037] Step A20: If the content type of the content to be explained is a static identifier type, then by performing image recognition on the content to be explained, the identifier description information is matched for the content to be explained in the target electronic instruction manual.

[0038] Step A30: If the content type of the content to be explained is a dynamic video type, then extract a preset number of first frame frames to be explained from the content to be explained.

[0039] Step A40: By performing image recognition on each of the first frames to be explained, the common content information to be explained for each of the first frames to be explained is obtained.

[0040] Step A50: Based on the information to be explained, match the video description information to the content to be explained in the target electronic instruction manual.

[0041] In this embodiment, it should be noted that the content to be explained is used to characterize the content waiting for the target electronic instruction manual to be explained on the current operation interface. The content type of the content to be explained includes static identifier type and dynamic video type. The static identifier type is used to characterize the content waiting for the target electronic instruction manual to be explained on the current operation interface at a preset time point. The dynamic video type is used to characterize the operating status of the current operation interface waiting for the target electronic instruction manual to be explained within a preset time period. The interface description information includes identifier description information and video description information. The identifier description information is used to describe the content to be explained of the static identifier type, and the video description information is used to describe the content to be explained of the dynamic identifier type. The preset time period is determined by the user. For example, in one implementable method, the target device is a washing machine, the preset time period is 3 seconds, and the content to be explained is that a certain indicator light on the current operation interface flashes continuously within 3 seconds. In this case, the video description information is used to explain the meaning of the continuous flashing of the indicator light.

[0042] Additionally, it should be noted that the first frame to be described is an internally encoded frame composed of at least one GOP (Group of Pictures) sequence in the content to be described. The preset number of the first frame to be described depends on the preset time period. The content to be described is used to characterize the changes of the preset operation interface within the preset time period. For example, in one implementable method, the preset operation interface of the target device includes a red light, and the content to be described is that the red light flashes 3 times within 3 seconds. The video description information describes the operating status of the target device within 3 seconds. In a specific scenario, the indicator light of the TV flashes for 3 seconds, which is interpreted in the electronic manual as "the TV is about to enter standby mode".

[0043] As an example, steps A10 to A50 include: obtaining the content to be explained corresponding to the current operation interface; if the content format of the content to be explained is detected to be an image format, then the content type of the content to be explained is determined to be a static identifier type, and the identifier description information is matched for the content to be explained in the target electronic manual by performing image recognition on the content to be explained, wherein the content format can be obtained by querying the content file extension of the content to be explained; if the content format of the content to be explained is detected to be a video format, then the content type of the content to be explained is determined to be a dynamic video type; if the content type of the content to be explained is a dynamic video type, then a preset number of first screen frames to be explained are extracted from the content to be explained; according to a preset image feature extraction model, the first image features to be explained for each of the first screen frames to be explained are extracted respectively, and the content information to be explained corresponding to each of the first screen frames to be explained is obtained by comparing the feature similarity of each of the first image features to be explained; using the content information to be explained as an index, the corresponding video description information is queried in the target electronic manual.

[0044] The step of matching corresponding identifier description information for the content to be described in the target electronic instruction manual by performing image recognition on the content to be described includes:

[0045] Step B10: Obtain the second screen frame to be explained corresponding to the content to be explained;

[0046] Step B20: Based on the preset image feature extraction model, perform image feature extraction on the second frame to be explained to obtain the identification and explanation image features;

[0047] Step B30: Based on the deviation between the label description image features and the preset label description image features, determine the label description information corresponding to the content to be described, wherein the preset label description image features are image description image features extracted from the target electronic manual by the preset feature extraction model.

[0048] In this embodiment, it should be noted that the second frame to be explained is an internally encoded frame consisting of at least one GOP sequence corresponding to the content to be explained. Since the current operation interface usually has multiple contents to be explained, the identification image feature is used to characterize the identification feature of a single content to be explained. The preset identification image feature is the image description image feature extracted from the target electronic manual by the preset feature extraction model.

[0049] As an example, steps B10 to B30 include: obtaining a second screen frame corresponding to the content to be explained; extracting image features from the second screen frame according to the preset image feature extraction model to obtain identification and explanation image features; determining the identification description information corresponding to the content to be explained based on the deviation between the identification and explanation image features and the preset identification and explanation image features, wherein the preset identification and explanation image features are image description image features extracted from the target electronic manual through the preset feature extraction model.

[0050] In one feasible approach, assuming the content to be described is an icon to be described, and the icon to be described includes an icon x and an icon y with high similarity, the icon description information corresponding to icon x in the target electronic manual is icon x1, and the icon description information corresponding to icon y is icon y1, wherein the image features of icon x1 and icon y1 are preset identification description image features. Therefore, when determining the icon x1 corresponding to icon x in the target electronic manual, it is necessary to consider the deviation between the image features of icon x and the image features of icon x1, as well as the deviation value between the image features of icon x and the image features of icon y1. Then, the icon description information of the icon with the smaller deviation value between icon x1 and icon y1 is used as the identification description information of icon x.

[0051] Step S30: Display the interface operation content corresponding to the interface description information based on the preset display interface.

[0052] In this embodiment, it should be noted that the preset display interface is set on the smart display device and is used to display the interface operation content corresponding to the current operation interface. The interface operation content is used to characterize the usage instructions of the current operation interface. For example, in one implementable method, it is assumed that the current operation interface is the interface corresponding to the vehicle system console captured by a mobile phone, wherein the usage instructions corresponding to the buttons of the current operation interface are the interface operation content.

[0053] As an example, step S30 includes: displaying the interface operation content corresponding to the interface description information based on a preset display interface.

[0054] The preset operation interface includes a first preset operation interface and a second preset operation interface. The step of displaying the interface operation content corresponding to the interface description information based on the preset display interface includes:

[0055] Step C10: If the current operation interface is the first preset operation interface, then by obtaining the control description information corresponding to the current operation interface, the preset display interface is controlled to display the first interface description content corresponding to the control description information.

[0056] Step C20: If the current operation interface is the second preset operation interface, then control the preset display interface to display the second interface description content corresponding to the interface description information.

[0057] In this embodiment, it should be noted that the first preset operation interface is used to represent the operation content of the target device that needs to be interactively displayed through human-computer interaction, and the second preset operation interface is used to represent the operation content of the target device that does not need to be interactively displayed through human-computer interaction. The control description information is used to describe the interface controls in the current operation interface. The interface controls can specifically be interface buttons, interface areas, and interface text, etc.

[0058] As an example, steps C10 to C20 include: if the current operation interface is the first preset operation interface, then obtain the control description information input by the user through the current operation interface, and control the preset display interface to display the first interface description content corresponding to the control description information; if the current operation interface is the second preset operation interface, then control the preset display interface to display the second interface description content corresponding to the interface description information.

[0059] The device control method based on the electronic manual, after the step of displaying the interface operation content corresponding to the interface description information based on the preset display interface, further includes:

[0060] Step D10: Detect whether the target device is a remote control device;

[0061] Step D20: If yes, then control the target device to perform the interface operation corresponding to the interface operation content.

[0062] In this embodiment, it should be noted that in the Internet of Things (IoT), devices manufactured by the same manufacturer can be linked for control. For example, in one feasible approach, assuming the IoT is a home internet, all devices in the home internet support remote control. Specifically, the devices in the home internet include televisions, refrigerators, washing machines, and mobile phones from the same manufacturer. When a user takes a picture of the preset operation interface of the television with their mobile phone, the mobile phone uses an image recognition module to identify the interface description information and interface operation content corresponding to the preset operation interface. Since the television supports remote control, the mobile phone can directly generate corresponding operation instructions based on the interface operation content, thereby controlling the television to perform the corresponding interface operation.

[0063] Additionally, it should be noted that for devices in the Internet of Things that support remote control, they open up their own capabilities, allowing other devices to interface with and control the device's functions. Thus, when a device needs to demonstrate the normal operating status of the target device or handle abnormal operating status, it can execute the target device remotely.

[0064] As an example, steps D10 to D20 include: detecting whether the target device is a remote control device; if the target device is detected to be a remote control device, generating an interface operation command corresponding to the interface operation content, and controlling the target device to execute the interface operation corresponding to the interface operation command based on the interface operation command. Since when the target device is a remote control device, regardless of whether the operation instructions for the preset operation interface in the target electronic manual are troubleshooting instructions or normal demonstration instructions, the user can automatically remotely control the target device to perform the corresponding interface operation. Therefore, the user does not need to obtain the operation instructions for the preset operation interface from the preset display interface to achieve the purpose of operating the preset operation interface of the target device, thus improving the control effect of the target device.

[0065] Prior to the step of controlling the target device to execute the interface operation command corresponding to the interface operation content, the device control method based on the electronic manual further includes:

[0066] Step E10: Based on the interface description information, determine whether the target device is in an abnormal operating state;

[0067] Step E20: If yes, record the abnormal operation information of the target device;

[0068] Step E30: The abnormal operation information is sent to the device server so that the device server can generate interface operation commands, wherein the interface operation commands are used to control the target device to perform the interface operation corresponding to the interface operation content.

[0069] In this embodiment, it should be noted that the abnormal operating state is used to characterize the interface abnormality of the preset operation interface, and the abnormal operating information is used to characterize the interface information of the current operation interface in the abnormal operating state. Specifically, it can be the abnormal operating time, abnormal operating location, and abnormal operating phenomenon. For example, in one feasible approach, the fault can be identified by a timer. Assuming that the "reset" button of the target device continues to flash within the timer, the abnormal operating information is recorded.

[0070] Additionally, it should be noted that the device server is used to represent the server controlling different devices in the Internet of Things (IoT). The device server has plug-in extension capabilities. For different devices with preset operation interfaces, the IoT calls the corresponding function plug-ins to control different devices to execute the interface operations for the preset operation interfaces.

[0071] As an example, steps E10 to E30 include: if the interface description information is a first type of interface description information, then the target device is determined to be in normal operation; if the interface description information is a second type of interface description information, then the target device is determined to be in abnormal operation. The type of the interface description information is determined by the interface description content corresponding to the interface description information. When the interface description content contains information related to "abnormality," then the interface description information is used as the first type of interface description information. If the target device is in an abnormal state, then the abnormal operation information of the target display device is recorded. The abnormal operation information is sent to the device server so that the device server can generate interface operation commands, wherein the interface operation commands are used to control the target device to execute the interface operation corresponding to the interface operation content. Since the intelligent display device records the abnormal operation information of the target device when the target device is in an abnormal working state, and can then obtain the interface operation commands from the device server to execute the interface operation corresponding to the interface operation content through the abnormal operation information, that is, the intelligent display device has interface operation commands to control the target device to execute interface operations. Therefore, this lays the foundation for realizing remote control of the target device.

[0072] This application provides a device control method based on an electronic manual. Specifically, by determining the corresponding target electronic manual based on the device information of the target device, the method achieves the goal of matching the target device with the corresponding electronic manual. Through image recognition of the current operating interface of the target device, the method matches the interface description information corresponding to the current operating interface in the target electronic manual. Then, it displays the interface operation content corresponding to the interface description information on a preset display interface. Since the target electronic manual corresponding to the target device is obtained, image recognition of the current operating interface of the target device automatically matches the interface description information of the current operating interface, and then displays the interface operation content corresponding to the interface description information on the preset display interface. Because the preset display interface can intuitively reflect the interface description information of the current operating interface, the user can directly obtain the interface operation content of the current operating interface on the preset display interface. That is, it achieves the goal of automatically matching the current operating interface of the target device with the interface description information in the electronic manual, rather than requiring the user to perform a pairing process manually. Therefore, it overcomes the technical defect that users spend a lot of time searching for the corresponding interface operation content in the electronic manual, thus improving the efficiency of controlling the pairing of the electronic manual and the smart device's operating interface.

[0073] Example 2

[0074] Furthermore, referring to Figure 2 In another embodiment of this application, content that is the same as or similar to that in Embodiment 1 described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of controlling the target device to execute the interface operation corresponding to the interface operation content includes:

[0075] Step F10: Detect whether there is a first interface operation command corresponding to the interface operation content;

[0076] If so, in step F20, the target device is controlled to perform the corresponding interface operation according to the first interface operation command.

[0077] Step F30: If not, check whether the device server corresponding to the target device has a plug-in calling mechanism;

[0078] Step F40: If the device server has the plugin invocation mechanism, it invokes the corresponding second interface operation command through the plugin interface of the interface operation content to control the target device to perform the corresponding interface operation.

[0079] In this embodiment, it should be noted that, since the Internet of Things (IoT) typically contains devices from different manufacturers, i.e., devices with different server locations, direct remote control between devices is not possible. For example, in one feasible approach, assuming the IoT is a home internet, the home IoT devices include a television M from manufacturer J, a television N from manufacturer K, a mobile phone L from manufacturer J, and a computer O from manufacturer L. To control the television N via the mobile phone L, a plug-in mechanism must be invoked. That is, both the television N and the mobile phone L have plug-in interfaces for controlling a certain interface function. The mobile phone L can then send corresponding control commands to the television N through this plug-in interface. The plug-in mechanism is an executable program running on the system.

[0080] Additionally, it should be noted that both the first interface operation command and the second interface operation command are interface operation commands. The first interface operation command is used to characterize the interface operation command executed by the smart display device in remotely controlling the target device, and the second interface operation command is used to characterize the interface operation command executed by the smart display device in remotely controlling the target device.

[0081] As an example, steps F10 to F40 include: detecting whether a first interface operation command corresponding to the interface operation content exists; if a first interface operation command corresponding to the interface operation content is detected, then controlling the target device to perform the corresponding interface operation based on the first interface operation command; if a first interface operation command corresponding to the interface operation content is detected, then detecting whether the device server corresponding to the target device has a plug-in calling mechanism; if the device server has the plug-in calling mechanism, then calling the corresponding second interface operation command through the plug-in interface of the interface operation content to control the target device to perform the corresponding interface operation.

[0082] In one feasible approach, refer to Figure 3 , Figure 3To illustrate a scenario for interface operation and control of home IoT devices, this application scenario includes a smartphone 11, a smart TV 12, a smart TV 13, a computer 14, and a smart air conditioner 15. Each device supports remote control. The smartphone 11, smart TV 12, smart computer 13, and smart TV 14 are on the same local area network. Smart TV 12 and computer 14 are from manufacturer O, while smart TV 13 and smart air conditioner 15 are from manufacturer P. Device server 16 can control the preset operation interfaces of smart TV 12 and computer 13 to perform corresponding interface operations. Device server 17 can control the preset operation interfaces of smart TV 13 and smart air conditioner 15 to perform corresponding interface operations. In terms of interface operation, when a user takes a picture of the preset operation interface of the smart TV 12 with their mobile phone 11, the preset display interface of the smartphone 11 will display the interface operation content of the preset operation interface of the smart TV 12. If the interface operation content indicates that the target device is in an abnormal operating state, the smartphone can remotely control the smart TV 12 through the interface operation command sent by the device server 16. When a user takes a picture of the preset operation interface of the smart air conditioner 15 with their mobile phone 11, if they want to remotely control the smart air conditioner 15 through the interface operation command sent by the device server 17, the device server 17 needs to have a plug-in calling mechanism to open the ability to control the smart air conditioner 15 to the smartphone 11.

[0083] This application provides a device interface operation control method, which involves detecting whether a first interface operation command corresponding to the interface operation content exists; if so, controlling the target device to execute the corresponding interface operation based on the first interface operation command; if not, detecting whether the device server corresponding to the target device has a plug-in calling mechanism; if the device server has the plug-in calling mechanism, calling the corresponding second interface operation command through the plug-in interface of the interface operation content to control the target device to execute the corresponding interface operation. Compared to the method where interface operation commands can be obtained through the device servers of the smart display device and the target device only when the smart display device and the target device are from the same manufacturer, and then the target device can be remotely controlled on the smart display device based on the interface operation commands, this embodiment utilizes the plug-in calling mechanism to open up the ability of different manufacturers to control devices. Therefore, even if the target device and the smart display device are from different manufacturers, the purpose of controlling the preset operation interface of the target device on the smart display device can still be achieved, thus improving the convenience of the smart display device to control the interface operation of the target device.

[0084] Example 3

[0085] This application also provides a device control device based on an electronic instruction manual, see below. Figure 4 The device control device based on the electronic instruction manual includes:

[0086] The determination module 101 is used to determine the corresponding target electronic instruction manual based on the device information of the target device;

[0087] Matching module 102 is used to match the interface description information corresponding to the current operating interface in the target electronic manual by performing image recognition on the current operating interface of the target device;

[0088] The control module 103 is used to display the interface operation content corresponding to the interface description information based on the preset display interface.

[0089] Optionally, the preset operation interface includes a first preset operation interface and a second preset operation interface, and the control module 102 is further used for:

[0090] If the current operation interface is the first preset operation interface, then by obtaining the control description information corresponding to the current operation interface, the preset display interface is controlled to display the first interface description content corresponding to the control description information;

[0091] If the current operation interface is the second preset operation interface, then the preset display interface is controlled to display the second interface description content corresponding to the interface description information.

[0092] Optionally, the interface description information includes identification description information and video description information, and the control module 102 is further configured to:

[0093] Obtain the content to be explained corresponding to the current operation interface;

[0094] If the content to be explained is indeed content to be explained, then by performing image recognition on the content to be explained, the identification description information is matched for the content to be explained in the target electronic instruction manual;

[0095] If the content to be explained is content to be explained, then a preset number of first screen frames to be explained are extracted from the content to be explained;

[0096] By performing image recognition on each of the first screen frames to be explained, the common content information to be explained for each of the first screen frames to be explained is obtained;

[0097] Based on the information to be explained, the video description information is matched to the information to be explained in the target electronic instruction manual.

[0098] Optionally, the control module 102 is further configured to:

[0099] Obtain the second screen frame to be explained corresponding to the content to be explained;

[0100] Based on the preset image feature extraction model, image features are extracted from the second frame to be described to obtain the identification and description image features;

[0101] Based on the deviation between the specified label description image features and the preset label description image features, the label description information corresponding to the content to be described is determined, wherein the preset label description image features are image description image features extracted from the target electronic manual through the preset feature extraction model.

[0102] Optionally, the device control device based on the electronic instruction manual is further used for:

[0103] Detect whether the target device is a remote control device;

[0104] If so, then control the target device to perform the interface operation corresponding to the interface operation content.

[0105] Optionally, the device control device based on the electronic instruction manual is further used for:

[0106] Detect whether there is a first interface operation command corresponding to the interface operation content;

[0107] If so, the target device is controlled to perform the corresponding interface operation according to the first interface operation command;

[0108] If not, then check whether the device server corresponding to the target device has a plugin invocation mechanism;

[0109] If the device server has the plugin invocation mechanism, it can invoke the corresponding second interface operation command through the plugin interface of the interface operation content to control the target device to perform the corresponding interface operation.

[0110] Optionally, the device control device based on the electronic instruction manual is further used for:

[0111] Based on the interface description information, determine whether the target device is in an abnormal operating state;

[0112] If so, record the abnormal operation information of the target device;

[0113] The abnormal operation information is sent to the device server so that the device server can generate interface operation commands, wherein the interface operation commands are used to control the target device to perform the interface operation corresponding to the interface operation content.

[0114] The device control device based on an electronic instruction manual provided by this invention, employing the device control method based on an electronic instruction manual in the above embodiments, solves the technical problem of low efficiency in pairing the electronic instruction manual and the smart device's operating interface. Compared with the prior art, the beneficial effects of the device control device based on an electronic instruction manual provided by this invention are the same as those of the device control method based on an electronic instruction manual provided in the above embodiments, and other technical features in this device control device based on an electronic instruction manual are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0115] Example 4

[0116] This invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the device control method based on the electronic manual described in Embodiment 1 above.

[0117] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0118] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus.

[0119] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0120] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0121] The electronic device provided by this invention employs the device control method based on an electronic instruction manual as described in the above embodiments, solving the technical problem of low efficiency in pairing the electronic instruction manual and the smart device's operating interface. Compared with the prior art, the beneficial effects of the electronic device provided by this invention are the same as those of the device control method based on an electronic instruction manual provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0122] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0124] Example 5

[0125] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the device control method based on the electronic instruction manual in the above embodiment.

[0126] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0127] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0128] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the electronic device to: determine the corresponding target electronic instruction manual based on the device information of the target device; match the interface description information corresponding to the current operating interface in the target electronic instruction manual by performing image recognition on the current operating interface of the target device; and display the interface operation content corresponding to the interface description information based on a preset display interface.

[0129] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0132] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the aforementioned device control method based on an electronic instruction manual, thus solving the technical problem of low efficiency in controlling the pairing of the electronic instruction manual and the smart device's operating interface. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this invention are the same as those of the device control method based on an electronic instruction manual provided in the above embodiments, and will not be repeated here.

[0133] Example 6

[0134] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the device control method based on the electronic instruction manual described above.

[0135] The computer program product provided in this application solves the technical problem of low efficiency in pairing electronic instruction manuals and smart device operating interfaces. Compared with the prior art, the beneficial effects of the computer program product provided in this embodiment are the same as those of the device control method based on electronic instruction manuals provided in the above embodiments, and will not be repeated here.

[0136] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A device control method based on an electronic instruction manual, characterized in that, The device control method based on electronic manual includes: Based on the equipment information of the target device, determine the corresponding target electronic instruction manual, wherein the equipment information includes the equipment model, product logo and device nameplate information; By performing image recognition on the current operating interface of the target device, the interface description information corresponding to the current operating interface is matched in the target electronic instruction manual. The interface description information includes identifier description information and video description information. The content to be explained corresponding to the current operating interface is then obtained. If the content type of the content to be explained is a static identifier type, the identifier description information is matched in the target electronic instruction manual by performing image recognition on the content to be explained. If the content to be explained is a dynamic video type, a preset number of first frames to be explained are extracted from the content to be explained. By performing image recognition on each of the first frames to be explained, the content information to be explained corresponding to each of the first frames to be explained is obtained, and the video description information is matched in the target electronic instruction manual for the content information to be explained. The interface operation content corresponding to the interface description information is displayed based on the preset display interface; Before controlling the target device to execute the interface operation corresponding to the interface operation content, it is determined whether the target device is in an abnormal operating state based on the interface description information; if so, the abnormal operating information of the target device is recorded; the abnormal operating information is sent to the device server so that the device server can generate interface operation commands. Detect whether the target device is a remote control device; If so, then control the target device to execute the interface operation corresponding to the interface operation content, wherein the step of controlling the target device to execute the interface operation corresponding to the interface operation content includes: Detect whether a first interface operation command corresponding to the interface operation content exists; If so, then according to the first interface operation command, control the target device to perform the corresponding interface operation; If not, then check whether the device server corresponding to the target device has a plugin invocation mechanism; If the device server has the plugin invocation mechanism, it can invoke the corresponding second interface operation command through the plugin interface of the interface operation content to control the target device to perform the corresponding interface operation.

2. The device control method based on an electronic instruction manual as described in claim 1, characterized in that, The preset user interfaces include a first preset user interface and a second preset user interface. The step of displaying the interface operation content corresponding to the interface description information based on a preset display interface includes: If the current operation interface is the first preset operation interface, then by obtaining the control description information corresponding to the current operation interface, the first interface operation content adapted by the control description information is displayed on the preset display interface. If the current operation interface is the second preset operation interface, then the second interface operation content adapted to the interface description information is displayed on the preset display interface.

3. The device control method based on an electronic instruction manual as described in claim 1, characterized in that, The step of performing image recognition on the content to be explained and matching corresponding identifier description information for the content to be explained in the target electronic instruction manual includes: Obtain the second screen frame to be explained corresponding to the content to be explained; Based on the preset image feature extraction model, image features are extracted from the second frame to be described to obtain the identification and description image features; Based on the deviation between the specified label description image features and the preset label description image features, the label description information corresponding to the content to be described is determined, wherein the preset label description image features are image description image features extracted from the target electronic manual through the preset feature extraction model.

4. A device control device based on an electronic instruction manual, characterized in that, The device control device based on the electronic instruction manual includes: The determination module is used to determine the corresponding target electronic instruction manual based on the device information of the target device, wherein the device information includes the device model, product logo and device nameplate information; The matching module is used to perform image recognition on the current operating interface of the target device and match the interface description information corresponding to the current operating interface in the target electronic manual. The interface description information includes identifier description information and video description information. This allows for the acquisition of content to be explained corresponding to the current operating interface. If the content to be explained is a static identifier type, the identifier description information is matched to the content to be explained in the target electronic manual by performing image recognition on the content to be explained. If the content to be explained is a dynamic video type, a preset number of first frames to be explained are extracted from the content to be explained. Image recognition is then performed on each of the first frames to be explained to obtain the content to be explained information commonly corresponding to each of the first frames to be explained, and the video description information is matched to the content to be explained in the target electronic manual. The control module is used to display the interface operation content corresponding to the interface description information based on the preset display interface; The device control device based on the electronic instruction manual further includes: detecting whether the target device is a remote control device; if so, detecting whether there is a first interface operation command corresponding to the interface operation content; if so, controlling the target device to perform the corresponding interface operation according to the first interface operation command; if not, detecting whether the device server corresponding to the target device has a plug-in calling mechanism; if the device server has the plug-in calling mechanism, controlling the target device to perform the corresponding interface operation by calling the corresponding second interface operation command through the plug-in interface of the interface operation content. The device control device based on the electronic instruction manual further includes: determining whether the target device is in an abnormal operating state based on the interface description information; if so, recording the abnormal operating information of the target device; and sending the abnormal operating information to the device server so that the device server can generate interface operation commands.

5. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the device control method based on the electronic instruction manual as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program implementing a device control method based on an electronic instruction manual, the program being executed by a processor to implement the steps of the device control method based on an electronic instruction manual as described in any one of claims 1 to 3.

Images