Equipment control method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an equipment control method and device, equipment and a storage medium, a control page is displayed on terminal equipment, each functional component is displayed on the control page, and the method comprises the following steps: acquiring a control panel image of the equipment, and identifying the control panel image to obtain control panel information; the control panel information comprises control panel function information and layout information of each control panel function; according to the layout information of each control panel function, associating each functional component on the equipment control page with the control panel function information; and presenting each functional component which is the same as the layout of each control panel function on the control page. By associating the control panel function to each functional component of the control page, the control page is converted into the page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the control panel and the whole machine panel are consistent, and the phenomenon of mistaken touch caused by inconsistency of the operation layout of the displayed control page and the whole machine panel is avoided.

Description

Technical Field

[0001] The present invention relates to the field of equipment control technology, and in particular to an equipment control method, an equipment control device, a corresponding electronic device, and a corresponding computer-readable storage medium. Background Art

[0002] With the development of intelligent home appliances, the application of smart home appliances has gradually become popular. The application of smart home is manifested in controlling home appliances through mobile terminal APP (Application), realizing refined control and remote operation of home appliances on APP, which is suitable for more usage scenarios.

[0003] However, the layout of the control page of the smart home related APP is different from the layout of the operation panel when the user actually uses the whole machine. Although some specific groups (such as the middle-aged and elderly people) are familiar with the operation of the whole machine, they are not very adaptable to the control page with a different layout, which makes it easy to have problems such as accidental touch, which is not conducive to device control. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a device control method, a device control apparatus, a corresponding electronic device and a corresponding computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0005] An embodiment of the present invention discloses a device control method, wherein a control page is displayed on a terminal device, and each functional component is displayed on the control page. The method includes:

[0006] Acquire a control panel image of the device, identify the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function;

[0007] According to the layout information of the control panel functions, associating the functional components on the device control page with the control panel function information;

[0008] The control page presents various functional components that are identical to the layout of various control panel functions.

[0009] In an embodiment of the present invention, by associating the control panel functions with the various functional components of the control page, the control page is converted into a page layout that is consistent with the operation of the whole machine panel, so that the display, layout, and operation of the two are consistent, and the whole machine panel can be presented on the control page and provide the user with operation capabilities, thereby avoiding false touches caused by inconsistency between the displayed control page operation layout and the whole machine panel.

[0010] Optionally, a control interface of the first theme is displayed on the device control page, and the method further includes:

[0011] In response to a trigger instruction of a theme switching control on the device control page, displaying a theme switching page on the device control page;

[0012] In response to a trigger instruction of a theme creation control on the theme switching page, a shooting component of the terminal device is started to obtain a control panel image of the device, and the control panel image of the device is used to create a control interface of the second theme.

[0013] Optionally, the control interface of the second theme is a newly established control interface; or is converted from the control interface of the first theme.

[0014] Optionally, identifying the control panel image to obtain control panel function information includes:

[0015] Get a pre-trained image recognition model;

[0016] The control panel image is input into the image recognition model to obtain control panel function information corresponding to the real function components in the control panel image.

[0017] Optionally, the step of inputting the control panel image into the image recognition model to obtain control panel function information corresponding to the functional components in the control panel image includes:

[0018] Recognize the control panel image by the image recognition model to obtain a target pre-stored image;

[0019] The target real function component associated with the target pre-stored image is used as the control panel function information of the control panel image.

[0020] Optionally, the step of associating the target real function component with the target pre-stored image as the control panel function information of the control panel image includes:

[0021] Acquire tag information of the target pre-stored image, where the tag information is used to indicate a target real functional component associated with the target pre-stored image;

[0022] The control panel image is identified by using the tag information, and the target real function component is used as the control panel function information of the control panel image.

[0023] Optionally, the training process of the image recognition model includes the following:

[0024] Get several pre-stored images of different panel functions;

[0025] Using different label information to identify the pre-stored images of each panel function, wherein different label information is used to identify correspondingly associated different real functional components;

[0026] The image recognition model is trained using pre-stored images associated with corresponding real functional components.

[0027] Optionally, associating each functional component on the device control page with the control panel function information according to the layout information of each control panel function includes:

[0028] When the real function component indicated by the control panel function information matches the target function component on the device control page, the real function component and the corresponding layout information are mapped to the target function component.

[0029] Optionally, mapping the real functional component and the corresponding layout information to the target functional component includes:

[0030] Obtaining the function value of the real function component;

[0031] The function value is applied to the target function component according to the corresponding layout information.

[0032] The embodiment of the present invention further discloses a device control apparatus, which displays a control page on a terminal device, and displays various functional components on the control page. The device includes:

[0033] A control panel image recognition module, used to obtain a control panel image of the device, recognize the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function;

[0034] A functional component associating module, used to associate each functional component on the device control page with the control panel function information according to the layout information of each control panel function;

[0035] The control page presenting module is used to present various functional components on the control page that are identical to the layout of various control panel functions.

[0036] In an embodiment of the present invention, by associating the control panel functions with the various functional components of the control page, the control page is converted into a page layout that is consistent with the operation of the whole machine panel, so that the display, layout, and operation of the two are consistent, and the whole machine panel can be presented on the control page and provide the user with operation capabilities, thereby avoiding false touches caused by inconsistency between the displayed control page operation layout and the whole machine panel.

[0037] Optionally, a control interface of the first theme is displayed on the device control page, and the apparatus further includes:

[0038] A control panel image acquisition module is used to respond to a trigger instruction of a theme switching control on the device control page, and display a theme switching page on the device control page; respond to a trigger instruction of a theme creation control on the theme switching page, start the shooting component of the terminal device, and obtain a control panel image of the device, and the control panel image of the device is used to create a control interface for a second theme.

[0039] Optionally, the control interface of the second theme is a newly established control interface; or is converted from the control interface of the first theme.

[0040] Optionally, the control panel image recognition module includes:

[0041] The control panel function recognition submodule is used to obtain a pre-trained image recognition model; input the control panel image into the image recognition model to obtain control panel function information corresponding to the real functional components in the control panel image.

[0042] Optionally, the control panel function identification submodule includes:

[0043] The control panel function recognition unit is used to recognize the control panel image through the image recognition model to obtain a target pre-stored image; and use the target real function component associated with the target pre-stored image as the control panel function information of the control panel image.

[0044] Optionally, the control panel function identification unit includes:

[0045] The control panel function identification subunit is used to obtain the label information of the target pre-stored image, where the label information is used to indicate the target real function component associated with the target pre-stored image; use the label information to identify the control panel image, and use the target real function component as the control panel function information of the control panel image.

[0046] Optionally, the device further comprises:

[0047] The image recognition model training module is used to obtain several pre-stored images of different panel functions; use different label information to identify the pre-stored images of each panel function, wherein different label information is used to identify different corresponding associated real functional components; use pre-stored images associated with corresponding real functional components to train an image recognition model.

[0048] Optionally, the functional component association module includes:

[0049] The function component association submodule is used to map the real function component and the corresponding layout information to the target function component when the real function component indicated by the control panel function information matches the target function component on the device control page.

[0050] Optionally, the functional component association submodule includes:

[0051] The functional component association unit is used to obtain the functional value of the real functional component; and apply the functional value to the target functional component according to the corresponding layout information.

[0052] An embodiment of the present invention further discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements any one of the device control methods when executed by the processor.

[0053] The embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any one of the device control methods is implemented.

[0054] The embodiments of the present invention include the following advantages:

[0055] In an embodiment of the present invention, the display control page provided by the terminal device displays various functional components, and the control panel information can be obtained by identifying the control panel image. According to the layout information of each control panel function, the various functional components on the device control page are associated with the control panel function information, and then the various functional components with the same layout as the various control panel functions are presented on the control page, so as to realize the conversion of the control page to a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the operation capability can be provided to the user, thereby avoiding the accidental touch phenomenon caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page to a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific people to control home appliances. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flowchart of steps of an embodiment of a device control method of the present invention;

[0057] Figure 2 is a flowchart of steps of another device control method embodiment of the present invention;

[0058] FIG. 3A to FIG. 3D is a page schematic diagram of a control page provided by an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of the image recognition process provided by an embodiment of the present invention;

[0060] Figure 5 It is a structural block diagram of an embodiment of a device control device of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] The embodiment of the present invention converts the control page into a page layout consistent with the whole machine panel operation by associating the control panel function with each functional component of the control page, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the user can be provided with operation capabilities, thereby avoiding the phenomenon of accidental touch caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page into a page layout consistent with the whole machine panel operation can reduce the user's learning cost, which is beneficial for some specific groups of people to control home appliances.

[0063] Reference Figure 1 , shows a flow chart of a device control method embodiment of the present invention, which may specifically include the following steps:

[0064] Step 101, acquiring a control panel image of a device, identifying the control panel image to obtain control panel information;

[0065] The application of smart home is to control home appliances through mobile terminal APP, and realize refined control and remote operation of home appliances on APP. Specifically, the terminal device can display a control page for controlling the device, and various functional components are displayed on the control page. Each functional component can be used to control different control functions of the device respectively.

[0066] In order to convert the control page into a page layout that is consistent with the operation of the entire machine panel, that is, to ensure that the display, layout, and operation of the various functional components displayed on the control page are consistent with the display, layout, and operation of the control panel of the entire device, the control panel image of the device can be obtained, and image recognition can be performed on the control panel image to obtain the control panel information, so as to adjust the layout of the various functional components on the control page based on the control panel information.

[0067] Optionally, the identified control panel information may include control panel function information and layout information of each control panel function. The control panel function information may refer to the text content of the control panel function, and the identified text content may be used to indicate the corresponding real function components on the control panel. The real function components may be represented as real buttons, controls, etc. on the control panel, which is not limited in the embodiment of the present invention; the layout information of the control panel function may indicate the specific location of the real function components corresponding to the control panel function on the control panel.

[0068] Step 102, according to the layout information of each control panel function, associate each functional component on the device control page with the control panel function information;

[0069] In one embodiment of the present invention, in order to convert the control page into a page layout that is consistent with the operation of the whole machine panel so that the display, layout, and operation of the two are consistent, the various functional components on the device control page can be associated with the control panel function information to achieve consistency in operation logic; and when associating the various functional components with the control panel function information, they can also be associated with the layout information of each control panel function to achieve consistency in display and layout.

[0070] Step 103 , presenting various functional components having the same layout as the various control panel functions on the control page.

[0071] After associating each functional component on the device control page with the control panel function information to achieve consistent operating logic; and, in combination with the layout information of each control panel function, associating each functional component with the control panel function information to achieve consistent display and layout, each functional component identical to the layout of each control panel function can be presented on the control page, thereby converting the control page to a page layout consistent with the operation of the entire machine panel.

[0072] After presenting various functional components with the same layout as the functions of each control panel on the control page, the corresponding device functions can be manipulated in response to the user's triggering operations on the various functional components presented on the control page. At this time, when the user triggers the various functional components on the control page, he can implement it according to the original control habits of the control panel of the entire device.

[0073] In an embodiment of the present invention, the display control page provided by the terminal device displays various functional components, and the control panel information can be obtained by identifying the control panel image. According to the layout information of each control panel function, the various functional components on the device control page are associated with the control panel function information, and then the various functional components with the same layout as the various control panel functions are presented on the control page, so as to realize the conversion of the control page to a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the operation capability can be provided to the user, thereby avoiding the accidental touch phenomenon caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page to a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific people to control home appliances.

[0074] Reference Figure 2 , shows a flow chart of another device control method embodiment of the present invention, which may specifically include the following steps:

[0075] Step 201, a control interface of the first theme is displayed on the device control page;

[0076] The application of smart home is to control home appliances through mobile terminal APP, and realize refined control and remote operation of home appliances on APP. Specifically, a control page for controlling the device can be displayed on the terminal device, and a control interface can be provided on the control page. Various functional components can be displayed on the control interface, and each functional component can be used to control different control functions of the device respectively.

[0077] In actual applications, the control page can provide control interfaces with different themes. In control interfaces with different themes, the layout display and control style settings of various functional components are different. For example, the first theme is the ring theme, and the controlled device is an air conditioner. Figure 3A As shown, in the control interface of the ring theme, various functional components, such as temperature adjustment +, temperature adjustment -, shutdown, mode, wind speed and direction, timing, etc., can be presented in a ring layout, and the corresponding control style of each functional component can be circular, etc., which is not limited by the embodiments of the present invention.

[0078] Step 202, in response to a trigger instruction of a theme switching control on a device control page, displaying a theme switching page on the device control page;

[0079] Optional, such as Figure 3B As shown, in the control interface of each theme, a theme switching control can be provided to switch the theme so as to present a control interface with a different layout. Figure 3BAs shown, in the control interface of each theme, a device information control can also be provided to query the device corresponding to the control of the current control interface.

[0080] When the user has a theme switching requirement, the terminal device can respond to the user's trigger instruction on the theme switching control and display the following on the device control page: Figure 3C The displayed theme switching page may include thumbnail information of control interfaces showing multiple themes, and the thumbnail information of control interfaces of multiple themes allows the control interface of the corresponding theme to be jumped and displayed when triggered.

[0081] Optionally, in addition to providing a switch jump to an existing theme control interface, the theme switching page can also provide users with the ability to create a new theme, specifically by displaying a theme creation control on the theme switching page. The theme creation control allows the creation of a new theme control interface when triggered by the user, thereby creating a control interface with a new functional component layout. For example, Figure 3C As shown, the theme creation control may be presented as an entry for "Create My Theme", which is not limited in this embodiment of the present invention.

[0082] Optionally, when creating a control interface of a new theme, the device to be controlled can be set accordingly, that is, the device information of the currently created control interface can be set.

[0083] Step 203, in response to a trigger instruction of a theme creation control on a theme switching page, a shooting component of the terminal device is started to obtain an image of a control panel of the device;

[0084] When the user has a need to create a theme, for example, the user does not know how to use the control interface of the first theme to operate the device, the terminal device can respond to the user's trigger instruction on the theme creation control to create a new control interface for the new theme.

[0085] The control interface of the new theme can be represented by starting the shooting component of the terminal device to obtain the control panel image of the device, and the control panel image of the device can be used to create the control interface of the second theme. Optionally, the control interface of the second theme is a newly established control interface, or is converted based on the control interface of the first theme, which is not limited by the embodiment of the present invention.

[0086] In actual applications, you can start the shooting component of the terminal device to shoot the entire control panel of the device to be controlled. In order to ensure that the display, layout, and operation in the created control interface are consistent with the entire control panel, when shooting the entire control panel of the device to be controlled, try to put the entire control panel into the shooting frame as completely as possible.

[0087] Step 204, identifying the control panel image to obtain control panel information;

[0088] In order to convert the control page into a page layout that is consistent with the operation of the whole machine panel, that is, to achieve consistency in the display, layout, and operation of the various functional components displayed on the control page with the control panel of the entire device, the control panel image can be recognized to obtain the control panel information, so that the layout of the various functional components on the control page can be adjusted based on the control panel information, so that the display, layout, and operation of the two are consistent, the whole machine panel can be presented on the control page and the user can be provided with operation capabilities, thereby avoiding false touches caused by inconsistency between the displayed control page operation layout and the whole machine panel.

[0089] Optionally, the identified control panel information may include control panel function information and layout information of each control panel function. The control panel function information may refer to the text content of the control panel function, and the identified text content may be used to indicate the corresponding real function component on the control panel; the layout information of the control panel function may indicate the specific location of the real function component corresponding to the control panel function on the control panel.

[0090] In the embodiment of the present invention, the recognition of the control panel image can be achieved through a pre-trained image recognition model.

[0091] In practical applications, we can mainly obtain a pre-trained image recognition model and then perform the following Figure 4 In the process shown, the control panel image to be recognized is input into the image recognition model to obtain the output result.

[0092] The output result may refer to the control panel function information corresponding to the real function component in the control panel image. Specifically, the control panel image is recognized by the image recognition model to obtain a target pre-stored image, and then the target real function component associated with the target pre-stored image is used as the control panel function information of the control panel image.

[0093] The real functional component associated with the pre-stored image can be identified based on the label information. That is, when obtaining the target real functional component associated with the target pre-stored image, the label information of the target pre-stored image can be obtained. Since the label information can be used to indicate the target real functional component associated with the target pre-stored image, the control panel image can be identified by using the label information, and the target real functional component can be used as the control panel function information of the control panel image.

[0094] Specifically, the image recognition model can extract image features and match them with pre-stored image features, and output matching results, i.e., identified functional components.

[0095] Exemplarily, taking the controlled device as an air conditioner, it is assumed that a text image of the mode can be pre-stored in the model library of a specific model of the air conditioner, and the pre-stored text image is identified by label 1, and label 1 can be used to indicate that the aforementioned pre-stored text image is associated with a functional component actually used to control the air conditioner mode, such as a click mode; the control panel image to be identified is the input unlabeled picture, and the image recognition model can output a result that is most similar to the unlabeled picture. For example, based on the regional image of the indication mode on the control panel image, the pre-stored image of the indication mode that is most similar in the model library can be identified. At this time, the control panel image can be identified by label 1, so that the functional component actually used to control the air conditioner mode indicated by label 1 is used as the control panel function information of the control panel image.

[0096] In one embodiment of the present invention, the training process of the image recognition model can be expressed as obtaining several pre-stored images of different panel functions, and then using different label information to identify the pre-stored images of each panel function, wherein different label information is used to identify different corresponding associated real functional components, and then using pre-stored images associated with corresponding real functional components to train the image recognition model.

[0097] Specifically, first, a large amount of image data can be collected as training sets and test sets. The collected image data can be text images with different function names pre-stored in the model library. Then, the collected images can be pre-processed, including denoising, resizing, normalization and other operations to ensure that the input images have consistent features for subsequent processing. In selecting a suitable deep learning model, considering that convolutional neural networks (CNN, Convolutional Neural Network, is a deep learning model suitable for processing data with a grid structure) perform well in image recognition tasks, the convolutional neural network can be trained using the aforementioned training set. During the training process, the model will learn how to extract useful features from images and associate these features with corresponding labels (used to indicate the associated real functional components). Specifically, text images with different function names are pre-stored in the model library, and each image is labeled. The labels are associated with real functional components, so that each pre-stored image can be associated with a real functional component. For example, in the model library of a specific air conditioner model, text images of modes can be pre-stored. The pre-stored text images are identified by label 1. Label 1 can be used to indicate that the aforementioned pre-stored text images are associated with functional components that are actually used to control the air conditioner mode, such as click modes, including cooling, heating, dehumidification, air supply, automatic mode, etc. The aforementioned pre-stored text images can be used as training sample data for the neural network.

[0098] Optionally, in the process of feature extraction, convolutional layers, pooling layers, and fully connected layers are involved. Among them, the convolutional layer in CNN is responsible for extracting local features of the image, such as edges, textures, etc. The pooling layer can be used to reduce the dimension of the feature and reduce the amount of calculation while retaining important feature information. The fully connected layer can be used to combine the extracted features to form the final classification result. Optionally, in the process of training the neural network, weight adjustment is also involved. The weights in the model are a set of parameters in the network, which are used to determine how the model extracts features from the input data. The weights can be adjusted mainly through the back-propagation algorithm according to the training data so that the model's prediction results are as close as possible to the true labels.

[0099] Exemplarily, unlabeled training sample data can be put into a neural network for pre-training. The first layer of neurons can first extract simple shapes, such as edges. The second layer of high-level neurons can continue to extract more complex structures. The top layer of neurons can then extract highly complex and abstract concepts, distinguish the standards of different pre-stored text images, and then output the result that is most similar to the control panel image to be identified based on the trained capabilities of the neural network. For example, based on the regional image of the indication pattern on the control panel image, the pre-stored image of the indication pattern that is most similar to the model library is identified.

[0100] It should be noted that, in addition to using the above-mentioned trained image recognition model to recognize the text content on the control panel image, the image recognition technology can also use OCR (OCR, optical character recognition) is a technology that converts images of printed or handwritten text into machine-encoded text) for recognition. For example, after inputting the control panel image to be recognized and recognizing the control panel image, the area where the text exists in the control panel image can be located by text detection, and then the located text area can be recognized by text recognition, and then the text can be output, and the corresponding function is determined based on the output text, and the text area where the function text is located is obtained, and the original functional component on the control page is associated with the current text area, and then associated with the real functional component. In addition to the image recognition technology proposed in the embodiment of the present invention, other image recognition technologies can also be used, and the embodiment of the present invention is not limited to this.

[0101] Step 205, according to the layout information of each control panel function, associate each functional component on the device control page with the control panel function information;

[0102] In one embodiment of the present invention, in order to convert the control page into a page layout that is consistent with the operation of the whole machine panel so that the display, layout, and operation of the two are consistent, the various functional components on the device control page can be associated with the control panel function information to achieve consistency in operation logic; and when associating the various functional components with the control panel function information, they can also be associated with the layout information of each control panel function to achieve consistency in display and layout.

[0103] Specifically, when the real function component indicated by the control panel function information matches the target function component on the device control page, the real function component and the corresponding layout information can be mapped to the target function component. Specifically, the function value of the real function component can be obtained, and the function value can be applied to the target function component according to the corresponding layout information to achieve mapping of the target function component.

[0104] Among them, the control interface of the second theme can be converted based on the control interface of the first theme. The different functional components displayed on the control interface of the existing theme can be regarded as the preset of the skeuomorphic layout. After image recognition through the image recognition model, the real functional components associated with the same label at the time of output and the functional values ​​of the real functional components can be directly applied; and as for position association, it can be expressed as locating the original button position of a control panel function on the control panel through image recognition, and then realizing the specific position of the corresponding functional component displayed in the skeuomorphic manner on the control interface presented by the terminal device.

[0105] Taking the controlled device as an air conditioner, as an example, assuming that when a mode button is identified from the control panel image, the mode button has an association relationship with the real function component related to the mode. At this time, the target function component related to the mode control control displayed on the control page can be associated with the real function component related to the mode, and the aforementioned target function component is bound to the target real function component. After binding, the user can directly obtain the function value of the target real function component on the control interface. The function value can be expressed as a mode value, which can include cooling, heating, air supply, dehumidification, automatic mode, etc., and then the obtained mode value can be directly used in the control interface. Mapping on the target functional component; as another example, assuming that when a switch button is identified from the control panel image, the switch button has an association with the real functional component related to the switch. At this time, the target functional component related to the switch control control displayed on the control page can be associated with the real functional component related to the switch, and the aforementioned target functional component is bound to the target real functional component. After binding, the user can directly obtain the function value of the target real functional component on the control interface. The function value can be expressed as a switch value, which can include on, off, etc., and then the obtained switch value can be directly mapped to the target functional component. Referring to the above process, all control panel functions in the control panel image, including temperature adjustment, wind speed and direction, timing, etc., can be mapped to each functional component on the device control page in turn, and then the following can be obtained. Figure 3D As shown, the control interface is consistent with the layout of the whole machine panel operation.

[0106] Take the steam oven as an example, assuming that the user does not use the card layout operation that is different from the whole machine screen, at this time, the device terminal can display the theme switching page on the device control page by responding to the user's trigger instruction on the theme switching control on the device control page, and then create a new theme control interface for the steam oven by responding to the user's trigger instruction on the theme creation control on the theme switching page. Specifically, the shooting component of the terminal device can be started, and the whole machine screen of the steam oven can be completely placed in the shooting frame to capture the control panel image. At this time, the image recognition technology can be used to associate the function on the steam oven screen with the corresponding functional component under the card theme, and the function value can be directly applied to the functional component, and then the function on the steam oven screen can be automatically associated with the APP component to form the whole machine theme page, and the steam oven can be controlled by using the steam oven screen style instead of the card layout, reducing the user's learning cost and cognitive burden.

[0107] Step 206 , presenting various functional components having the same layout as the various control panel functions on the control page.

[0108] After associating each functional component on the device control page with the control panel function information to achieve consistent operating logic; and, in combination with the layout information of each control panel function, associating each functional component with the control panel function information to achieve consistent display and layout, each functional component identical to the layout of each control panel function can be presented on the control page, thereby converting the control page to a page layout consistent with the operation of the entire machine panel.

[0109] On the control page, the control interface presented with the same layout as the functions of each control panel may be the control interface of the second theme.

[0110] After presenting various functional components with the same layout as the functions of each control panel on the control page, the corresponding device functions can be manipulated in response to the user's triggering operations on the various functional components presented on the control page. At this time, when the user triggers the various functional components on the control page, he can implement it according to the original control habits of the control panel of the entire device.

[0111] For example, the generated Figure 3D The whole machine theme page shown in the figure can be the control interface of the second theme newly created as described above, so that Figure 3D The remote control style control interface shown can replace Figure 3C The circular-themed control interface shown allows users to control the air conditioner according to their original control habits of the device's control panel, reducing users' learning costs and cognitive burden.

[0112] It should be noted that the user can also switch back to the original ring theme control interface by using the theme switching control provided on the control interface, and this embodiment of the present invention does not limit this.

[0113] In the embodiment of the present invention, by associating the control panel function with each functional component of the control page, the control page is converted into a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the user can be provided with operation capabilities, thereby avoiding the phenomenon of accidental touch caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page into a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific groups of people to control home appliances.

[0114] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0115] Reference Figure 5 , shows a structural block diagram of an embodiment of a device control device of the present invention, wherein a control page is displayed on a terminal device, and each functional component is displayed on the control page, which may specifically include the following modules:

[0116] A control panel image recognition module 501 is used to obtain a control panel image of the device, recognize the control panel image and obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function;

[0117] A function component associating module 502, configured to associate each function component on the device control page with the control panel function information according to the layout information of each control panel function;

[0118] The control page presenting module 503 is used to present various functional components on the control page that are identical to the layout of various control panel functions.

[0119] In an embodiment of the present invention, by associating the control panel functions with the various functional components of the control page, the control page is converted into a page layout that is consistent with the operation of the whole machine panel, so that the display, layout, and operation of the two are consistent, and the whole machine panel can be presented on the control page and provide the user with operation capabilities, thereby avoiding false touches caused by inconsistency between the displayed control page operation layout and the whole machine panel.

[0120] In an embodiment of the present invention, the device control page displays a control interface of the first theme, and the apparatus provided by the embodiment of the present invention may further include the following modules:

[0121] A control panel image acquisition module is used to respond to a trigger instruction of a theme switching control on the device control page, and display a theme switching page on the device control page; respond to a trigger instruction of a theme creation control on the theme switching page, start the shooting component of the terminal device, and obtain a control panel image of the device, and the control panel image of the device is used to create a control interface for a second theme.

[0122] In an embodiment of the present invention, the control interface of the second theme is a newly established control interface; or is converted based on the control interface of the first theme.

[0123] In one embodiment of the present invention, the control panel image recognition module 501 may include the following submodules:

[0124] The control panel function recognition submodule is used to obtain a pre-trained image recognition model; input the control panel image into the image recognition model to obtain control panel function information corresponding to the real functional components in the control panel image.

[0125] In one embodiment of the present invention, the control panel function identification submodule may include the following units:

[0126] The control panel function recognition unit is used to recognize the control panel image through the image recognition model to obtain a target pre-stored image; and use the target real function component associated with the target pre-stored image as the control panel function information of the control panel image.

[0127] In one embodiment of the present invention, the control panel function identification unit may include the following subunits:

[0128] The control panel function identification subunit is used to obtain the label information of the target pre-stored image, where the label information is used to indicate the target real function component associated with the target pre-stored image; use the label information to identify the control panel image, and use the target real function component as the control panel function information of the control panel image.

[0129] In one embodiment of the present invention, the embodiment of the present invention may further include the following modules:

[0130] The image recognition model training module is used to obtain several pre-stored images of different panel functions; use different label information to identify the pre-stored images of each panel function, wherein different label information is used to identify different corresponding associated real functional components; use pre-stored images associated with corresponding real functional components to train an image recognition model.

[0131] In one embodiment of the present invention, the functional component association module 502 may include the following submodules:

[0132] The function component association submodule is used to map the real function component and the corresponding layout information to the target function component when the real function component indicated by the control panel function information matches the target function component on the device control page.

[0133] In one embodiment of the present invention, the functional component association submodule may include the following units:

[0134] The functional component association unit is used to obtain the functional value of the real functional component; and apply the functional value to the target functional component according to the corresponding layout information.

[0135] In an embodiment of the present invention, the display control page provided by the terminal device displays various functional components. The device control device provided by the embodiment of the present invention can obtain control panel information by identifying the control panel image, and associate various functional components on the device control page with the control panel function information according to the layout information of each control panel function, and then present various functional components with the same layout as each control panel function on the control page, so as to realize the conversion of the control page into a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the operation capability can be provided to the user, thereby avoiding the accidental touch phenomenon caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page into a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific people to control home appliances.

[0136] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] An embodiment of the present invention further provides an electronic device, including:

[0138] The device comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the device control method embodiment is implemented, specifically implementing the following steps:

[0139] Acquire a control panel image of the device, identify the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function;

[0140] According to the layout information of the control panel functions, associating the functional components on the device control page with the control panel function information;

[0141] The control page presents various functional components that are identical to the layout of various control panel functions.

[0142] In an embodiment of the present invention, various functional components are displayed on the display control page provided by the terminal device. The electronic device provided by the embodiment of the present invention can obtain control panel information by identifying the control panel image, and associate various functional components on the device control page with the control panel function information according to the layout information of each control panel function, and then present various functional components with the same layout as each control panel function on the control page, so as to realize the conversion of the control page to a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the operation capability can be provided to the user, thereby avoiding the accidental touch phenomenon caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page to a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific people to control home appliances.

[0143] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned device control method embodiment is implemented, specifically implementing the following steps:

[0144] Acquire a control panel image of the device, identify the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function;

[0145] According to the layout information of the control panel functions, associating the functional components on the device control page with the control panel function information;

[0146] The control page presents various functional components that are identical to the layout of various control panel functions.

[0147] In an embodiment of the present invention, various functional components are displayed on the display control page provided by the terminal device. The computer-readable storage medium provided by the embodiment of the present invention can obtain control panel information by identifying the control panel image, and associate various functional components on the device control page with the control panel function information according to the layout information of each control panel function, and then present various functional components with the same layout as each control panel function on the control page, so as to realize the conversion of the control page into a page layout consistent with the operation of the whole machine panel, so that the display, layout and operation of the two are consistent, and the whole machine panel can be presented on the control page and the operation capability can be provided to the user, thereby avoiding the phenomenon of false touch caused by the inconsistency between the displayed control page operation layout and the whole machine panel. Furthermore, converting the control page into a page layout consistent with the operation of the whole machine panel can reduce the user's learning cost, which is beneficial for some specific people to control home appliances.

[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0149] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0150] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0154] Finally, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0155] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0156] The above is a detailed introduction to a device control method, a device control apparatus, a corresponding electronic device and a corresponding computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A device control method, characterized in that: The method includes: displaying a control page on a terminal device, wherein each functional component is displayed on the control page; Acquire a control panel image of the device, identify the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function; According to the layout information of the control panel functions, associating the functional components on the device control page with the control panel function information; The control page presents various functional components that are identical to the layout of various control panel functions.

2. The method according to claim 1, characterized in that The device control page displays a control interface of the first theme, and the method further includes: In response to a trigger instruction of a theme switching control on the device control page, displaying a theme switching page on the device control page; In response to a trigger instruction of a theme creation control on the theme switching page, a shooting component of the terminal device is started to obtain a control panel image of the device, and the control panel image of the device is used to create a control interface of the second theme.

3. The method according to claim 2, characterized in that The control interface of the second theme is a newly established control interface; or is converted from the control interface of the first theme.

4. The method according to claim 1, characterized in that: Recognizing the control panel image to obtain control panel function information includes: Get a pre-trained image recognition model; The control panel image is input into the image recognition model to obtain control panel function information corresponding to the real function components in the control panel image.

5. The method according to claim 4, characterized in that The step of inputting the control panel image into the image recognition model to obtain control panel function information corresponding to the functional components in the control panel image includes: Recognize the control panel image by the image recognition model to obtain a target pre-stored image; The target real function component associated with the target pre-stored image is used as the control panel function information of the control panel image.

6. The method according to claim 5, characterized in that The target real function component associated with the target pre-stored image as the control panel function information of the control panel image includes: Acquire tag information of the target pre-stored image, where the tag information is used to indicate a target real functional component associated with the target pre-stored image; The control panel image is identified by using the tag information, and the target real function component is used as the control panel function information of the control panel image.

7. The method according to any one of claims 4 to 6, characterized in that: The training process of the image recognition model includes the following: Get several pre-stored images of different panel functions; Using different label information to identify the pre-stored images of each panel function, wherein different label information is used to identify correspondingly associated different real functional components; The image recognition model is trained using pre-stored images associated with corresponding real functional components.

8. The method according to claim 1, characterized in that The associating each functional component on the device control page with the control panel function information according to the layout information of each control panel function includes: When the real function component indicated by the control panel function information matches the target function component on the device control page, the real function component and the corresponding layout information are mapped to the target function component.

9. The method according to claim 8, characterized in that The mapping of the real functional component and the corresponding layout information to the target functional component includes: Obtaining the function value of the real function component; The function value is applied to the target function component according to the corresponding layout information.

10. A device control device, characterized in that: By displaying a control page on a terminal device, each functional component is displayed on the control page, and the device includes: A control panel image recognition module, used to obtain a control panel image of the device, recognize the control panel image to obtain control panel information; the control panel information includes control panel function information and layout information of each control panel function; A functional component associating module, used to associate each functional component on the device control page with the control panel function information according to the layout information of each control panel function; The control page presenting module is used to present various functional components on the control page that are identical to the layout of various control panel functions.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the device control method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the device control method according to any one of claims 1 to 9 is implemented.

Images