Data fusion method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data fusion method and device, electronic equipment and a storage medium. The method relates to the technical field of data fusion in the field of data processing. In the embodiment, the event code of the first event in the handle data is converted to obtain the second event, and then the first touch data is determined based on the second event; reading second touch data from the screen input equipment node; based on the reading sequence of the second event and the reading sequence of the third event, fusing the first touch data and the second touch data to obtain fused data; writing the fused data into a virtual input equipment node; the fusion data is read from the virtual input equipment node, and the rendered image is obtained and displayed based on the fusion data, so that the compatibility of the handle and the touch screen and the use performance of the handle can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of data fusion in the field of data processing, and more specifically, to a data fusion method, device, electronic device and storage medium. Background Art

[0002] Key mapping is a method of mapping the controller data of a handheld game console to the touch data of a touch screen. It is widely used in handheld game consoles and external controllers on the Android platform to use the controller to operate games or applications that only support touch.

[0003] Generally, the principle of button mapping is: first obtain the handle data, then convert the handle data into touch data, and perform image rendering based on the converted touch data and display the rendered image.

[0004] However, since the formats of the touch data obtained through key mapping and the touch data generated after the user performs a touch operation on the touch screen are different, the touch data obtained through key mapping and the touch data of the touch screen cannot be merged, which affects the compatibility of the handle and the touch screen, and thus reduces the performance of the handle. Summary of the invention

[0005] The embodiments of the present application provide a data fusion method, device, electronic device, and storage medium, which can improve the compatibility of a handle and a touch screen and the performance of the handle.

[0006] In a first aspect, an embodiment of the present application provides a data fusion method, including:

[0007] Reading handle data from a handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used to identify the key that triggers the first event, and the value of the first event is used to identify the key operation that triggers the first event;

[0008] Obtaining a second event by converting the event code of the first event into an event code for identifying a touch event;

[0009] determining first touch data based on the second event;

[0010] Reading second touch data from the screen input device node; the second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event;

[0011] Based on the reading order of the second event and the reading order of the third event, the first touch data and the second touch data are merged to obtain fused data;

[0012] Write the fused data into the virtual input device node;

[0013] The fused data is read from the virtual input device node, and a rendered image is obtained and displayed based on the fused data.

[0014] In a second aspect, an embodiment of the present application provides a data fusion device, including:

[0015] A first reading unit is used to read handle data from a handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used to identify the key that triggers the first event, and the value of the first event is used to identify the key operation that triggers the first event;

[0016] a conversion unit, configured to obtain a second event by converting the event code of the first event into an event code for identifying a touch event;

[0017] a determining unit, configured to determine first touch data based on the second event;

[0018] A second reading unit is used to read second touch data from the screen input device node; the second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event;

[0019] a fusion unit, configured to fuse the first touch data and the second touch data based on a reading order of the second event and a reading order of the third event to obtain fused data;

[0020] A writing unit, used for writing the fused data into the virtual input device node;

[0021] The rendering unit is used to read the fused data from the virtual input device node, and obtain and display a rendered image based on the fused data.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0023] a processor adapted to implement computer instructions; and,

[0024] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the method of the first aspect mentioned above.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the method of the first aspect involved above.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the first aspect mentioned above.

[0027] According to the data fusion method provided in the embodiment of the present application, the second event is obtained by converting the event code of the first event in the handle data, and then the first touch data is determined based on the second event; the second touch data is read from the screen input device node; based on the reading order of the second event and the reading order of the third event, the first touch data and the second touch data are fused to obtain fused data; the fused data is written into the virtual input device node; the fused data is read from the virtual input device node, and the rendered image is obtained and displayed based on the fused data, which can improve the compatibility of the handle and the touch screen and the use performance of the handle. For example, since the rendered image is obtained based on the fused data, when playing with the handle, even if the touch screen is accidentally touched, it will not cause the operation of the handle to fail, and will not cause errors in the game or application, thereby improving the use performance of the handle. For example, some games have some operations that are not convenient to use the handle or when the handle mapping points are not enough, it is necessary to perform touch operations on the touch screen. In this embodiment, by fusing the first touch data and the second touch data, the operation requirements of operating the handle and the touch screen at the same time can also be met, thereby improving the use performance of the handle.

[0028] In addition, the second event is obtained by converting the event code of the first event in the handle data, and then the first touch data is determined based on the second event. This is equivalent to converting the handle data into touch data from the system layer, which can ensure the compatibility between the first touch data and the touch data of the touch screen, thereby improving the success rate of data fusion.

[0029] In addition, the second touch data is read from the screen input device node; based on the reading order of the second event and the reading order of the third event, the first touch data and the second touch data are fused to obtain fused data, which can avoid the conflict between the first touch data and the second touch data in timing and improve the fusion effect of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an example of a handheld device provided in an embodiment of the present application.

[0031] Figure 2 This is an example of a quick settings panel of the mapping configuration interface provided in an embodiment of the present application.

[0032] Figure 3 This is an example of the mapping configuration interface provided in an embodiment of the present application.

[0033] Figure 4 This is an example of a handheld game console interface provided in an embodiment of the present application.

[0034] Figure 5 It is a schematic flow chart of the data fusion method provided in the embodiment of the present application.

[0035] Figure 6 It is a schematic flowchart of the fusion data reading process provided in an embodiment of the present application.

[0036] Figure 7 This is an example of the data fusion process provided in the embodiments of the present application.

[0037] Figure 8 It is a schematic block diagram of a data fusion device provided in an embodiment of the present application.

[0038] Fig. 9 It is a schematic flow chart of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments provided by this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] To facilitate understanding of the technical solution provided by this application, the relevant terms are explained below.

[0041] Handheld game console: refers to a hardware device based on the Android platform that has a large number of touch-based games and applications. It is equipped with a game controller that can be used to control games. For example, a handheld game console can be a handheld portable game console or multimedia device, which usually has a built-in screen, processor, memory, operating system and other components. A handheld game console can run games, applications and media content, and users can use the handheld game console to play games, browse the Internet, watch videos and other functions.

[0042] Button mapping: A general technical name for mapping handheld controller input to screen touch input.

[0043] Input device node: In the Linux architecture, all input devices, such as screens, handles, and sensors, exist in the system bottom layer as input device nodes (a form of file). Input devices generate data in files, and the system reads the data in these files and provides them to system applications.

[0044] It should be noted that the terms used in the implementation method section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0045] For example, the term "and / or" in this article is only a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" is only a way to describe the combination relationship of listed objects, indicating that one or more items may exist. For example, at least one of the following: A, B, C can mean the following combinations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time. The term "multiple" means two or more. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0046] For another example, the term "corresponding" may indicate that there is a direct or indirect correspondence between the two, or that there is an association relationship between the two, or that there is an indication and being indicated, configuration and being configured, etc. The term "indication" may be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which may indicate that A directly indicates B, such as B can be obtained through A; it may also indicate that A indirectly indicates B, such as A indicates C, B can be obtained through C; it may also indicate that there is an association relationship between A and B. The term "predefined" or "preconfigured" may refer to the pre-storage of corresponding codes, tables or other relevant information that can be used for indication in the device, or it may refer to an agreement by protocol. "Protocol" may refer to a standard protocol in this field. The term "when..." may be interpreted as "if" or "if" or "when..." or "in response to" and other similar descriptions. Similarly, depending on the context, the phrase "if determined" or "if (stated condition or event) is detected" may be interpreted as "when determined" or "in response to determining" or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)" and similar descriptions. The terms "first", "second", "third", "fourth", "Ath", "Bth", etc. are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Among them, digital video compression technology mainly compresses huge digital image video data for easy transmission and storage.

[0047] The technical problem to be solved by this application is described in detail below.

[0048] Key mapping is a method of mapping the controller data of a handheld game console to the touch data of a touch screen. It is widely used in handheld game consoles and external controllers on the Android platform to use the controller to operate games or applications that only support touch.

[0049] Key mapping can be implemented in the following ways:

[0050] 1. Debug mode solution: The debug mode solution guides the user to turn on the debug mode to debug the handle. After the handheld game console reads the handle data, it converts the handle data into touch data and injects it into the device node using the debug mode turned on by the user, so that the handheld game console's system can read the touch data from the device node and achieve the operation effect of using the handle.

[0051] 2. Capacitive air mapping solution. The capacitive air mapping solution enables the handle to transmit capacitive signals through the air, thereby simulating the real touch screen operation effect of human hands.

[0052] 3. Application control scheme: specifically, an application (app) can be used to communicate directly with the handle via Bluetooth or Universal Serial Bus (USB), and touch data can be directly generated inside the handle by controlling the handle firmware.

[0053] However, since the debugging mode solution is usually a method of performing data conversion at the application level to implement a game or application that supports touch by using a handle operation, and the data generated after the user performs a touch operation on the touch screen is usually system-level data, the touch data obtained through key mapping and the touch data generated after the user performs a touch operation on the touch screen are in different formats, that is, the touch data obtained through key mapping and the touch data of the touch screen cannot be merged, which affects the compatibility of the handle and the touch screen, thereby reducing the performance of the handle. For example, when playing with a handle, if you accidentally touch the touch screen, the operation of the handle will fail, and even errors will occur in the game or application, thereby reducing the performance of the handle. For another example, when some operations in some games are not convenient to use the handle or the handle mapping points are not enough, the touch operation must be performed on the touch screen. However, since the touch data obtained through key mapping and the touch data of the touch screen cannot be merged, the handle will not meet the user's needs, thereby reducing the performance of the handle.

[0054] Similarly, since the capacitive signals sent by the capacitive air mapping solution and the touch signals generated inside the handle are not system-level data, they cannot be merged with the touch data of the touch screen, affecting the compatibility of the handle and the touch screen, and thus reducing the performance of the handle.

[0055] In view of this, the embodiments of the present application provide a data fusion method, device, electronic device and storage medium, which can improve the compatibility of a handle and a touch screen and the performance of the handle.

[0056] The solution provided in the present application is applicable to any portable device with a handle, for example, the portable device may be various handheld game consoles.

[0057] Figure 1 This is an example of a handheld device provided in an embodiment of the present application.

[0058] like Figure 1 As shown, the handheld game console may include a handle. The handle refers to a control device connected to a handheld game console or a game console. The handle usually includes a home button, a joystick, a direction key, a trigger and other control elements, which are used by the user to operate the game, application or handheld game console interface. The handle is connected to the handheld game console or the console to convert the user's input into corresponding signals and transmit them to the system.

[0059] In addition, the handheld game console may also include a touch screen, a processor, memory, storage, and an operating system.

[0060] Among them, the touch screen is a display device on the handheld game console, which usually uses liquid crystal or organic light-emitting diode (OLED) technology. It is used to display image information of games, applications and media content. The size, resolution and display technology of the handheld game console screen may vary. The processor is used to perform various computing tasks and instructions. The processor is responsible for the operation and data processing of the handheld game console, and plays an important role in the running speed and performance of games and applications. Memory is a memory component used to store data and temporary calculations, such as random access memory (RAM). The memory stores the games, applications and system data being executed by the handheld game console, and plays an important role in the multitasking and performance optimization of the handheld game console. Storage is a storage device such as a solid-state drive or a flash memory card for storing games, applications, media content and user data. The memory is a place on the handheld game console for long-term storage of data, and users can save games, applications and other files in it. The operating system can be a software system used by the handheld game console to manage hardware resources, run applications and provide a user interface. The operating system usually has functions such as a graphical user interface, application management, and storage management to enable users to operate the handheld game console conveniently.

[0061] By long pressing the home button of the controller, calling up the quick settings panel, and opening the key mapping panel on the quick settings panel, you can Figure 2 The setting interface shown in the figure includes the "Turn on (or off) key mapping" button, the "Show mapped keys" button, and the "Configure key mapping" button. By clicking the "Turn on (or off) key mapping" button, you can turn on or off the key mapping function, which can be used to convert the controller data into touch data. By clicking the "Show mapped keys" button, you can display the various keys of the controller in the mapping configuration interface. By clicking the "Configure key mapping" button, you can display Figure 3 The mapping configuration interface is shown in the figure. Figure 3 As shown, the mapping configuration interface may include mapping buttons for L2, L1, L3, L, R2, R1, R3, R, X, Y, A, B, etc., where L may be a left joystick button mapping button and R may be a right joystick button mapping button. The background of the mapping configuration interface may be a transparent background, which may float above the game / application interface (including the game screen and skill buttons) for easy viewing by users. Users can adjust the mapping configuration by dragging the mapping position, turning on / off the mapping button, adjusting the size, adjusting the mode, etc. Figure 3In the game, the gray button represents the mapping button, that is, the mapped buttons include L button, A button and B button. For example, the user can configure the left joystick to control the character's movement, the A button to control the character's skill 2 (such as kicking), and the B button to control the character's skill 1 (such as punching). The other buttons are in an inactive state. After saving the configuration, the user can control the game / application through the controller, and the display interface of the game / application can be as follows Figure 4 The interface shown.

[0062] The multi-scenario investigation method provided in the embodiment of the present application is described in detail below.

[0063] Figure 5 is a schematic flow chart of a multi-scenario investigation method 100 provided in an embodiment of the present application. The method 100 can be executed by any electronic device with data processing capability. For example, the electronic device can be implemented as any portable device with a handle. For example, the electronic device can be Figure 1 For the convenience of description, the following description is made by taking the data fusion device as the execution subject of the method 100 as an example.

[0064] like Figure 5 As shown, the method 100 may include:

[0065] S110, the data fusion device reads handle data from the handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used to identify the key that triggers the first event, and the value of the first event is used to identify the key operation that triggers the first event.

[0066] For example, the handle data refers to data generated by operating a handle or a controller. The handle data may include operation information of various buttons such as a start button, a joystick, a direction key, and a trigger.

[0067] It is worth noting that in the fields of computer programming and interactive design, data is usually represented by events used to represent interactive behaviors, and event types and event codes are important concepts used to describe and process events.

[0068] Event Type: Event type refers to the interactive action or triggered behavior performed by the user in the application or system. It can be a key press, mouse movement, touch screen operation, handle operation, etc. Different event types correspond to different object behaviors. Developers can identify and process corresponding user input based on event types.

[0069] Event Code: An event code is a numeric or string code used to identify a specific event type. Each event type usually has a specific set of event codes to represent different user interaction behaviors. For example, the event code for a left-click event on a mouse may be 1, and the event code for a keyboard key event may be encoded according to the layout of the keys on the keyboard. Through event codes, developers can identify and process specific user input to perform corresponding actions or trigger corresponding functions.

[0070] Event types and event codes are terms used in software development to capture and respond to user interactions. By detecting and handling different event types and event codes, developers can implement interactions with users, enabling applications to change behavior or present results based on user input. This interactivity can be applied to a variety of applications, including games, user interfaces, web pages, and mobile apps.

[0071] Exemplarily, the event type may include the following three types:

[0072] 1. EV_KEY: indicates a key event. When a user presses or releases a key on a keyboard, mouse, or other input device, an EV_KEY event is generated. Each key has a unique key code corresponding to it. The EV_KEY event can be used to obtain the key status (pressed or released) and the corresponding key code.

[0073] 2. EV_ABS: indicates a position event (which can also be called an absolute position event, an absolute input event, an absolute coordinate event, etc.). When an absolute position input device (such as a touch screen) is used, an EV_ABS event is generated. The EV_ABS event provides absolute position information on the input device, such as the coordinates of the touch point, touch pressure, etc.

[0074] 3. EV_SYN: indicates a synchronization event. The EV_SYN event is used to synchronize the input event stream. It inserts a synchronization point between different parts of the input event sequence to ensure the order and synchronization of events. The EV_SYN event is usually used to mark the beginning and end of the input event sequence.

[0075] In summary, EV_KEY is used to indicate key events, EV_ABS is used to indicate absolute position events, and EV_SYN is used to synchronize input event streams. They are used for different types of input devices and events respectively.

[0076] For example, for a touch operation, the event code of a key event may include the following two:

[0077] 1. BTN_TOUCH: Indicates the touch status of a touch device. When a finger touches a touch device, a BTN_TOUCH event is triggered, indicating that the device is being touched. When the finger leaves the touch device, the BTN_TOUCH event is released, indicating that the device is no longer being touched.

[0078] 2. BTN_TOOL_FINGER: Indicates the tool type of the touch device. When the touch device is touched by a finger, the BTN_TOOL_FINGER event is triggered, indicating that the tool type is a finger. Other tool types, such as a stylus or palm, may be represented by different BTN_TOOLs.

[0079] For example, for a handle operation, the event code of a key event may include the following:

[0080] 1. Event codes related to direction keys, including but not limited to:

[0081] BTN_DPAD_UP: Up direction button.

[0082] BTN_DPAD_DOWN: Down direction button.

[0083] BTN_DPAD_LEFT: Left direction button.

[0084] BTN_DPAD_RIGHT: Right direction button.

[0085] 2. Event codes related to joystick buttons, including but not limited to:

[0086] BTN_THUMBL: Left joystick button.

[0087] BTN_THUMBR: Right joystick button.

[0088] 3. Event codes related to function keys, including but not limited to:

[0089] BTN_START: Start button.

[0090] BTN_SELECT: Select button.

[0091] BTN_A: A button.

[0092] BTN_B: B button.

[0093] BTN_X: X button.

[0094] BTN_Y: Y button.

[0095] BTN_TRIGGER: Trigger button.

[0096] It should be understood that, for handle operation, the event code of the key event is used to identify different keys on the handle. Which event codes are used specifically depends on the support of the handle device and the driver, so there may be differences in actual use, and this application does not make specific limitations on this. For example, in other alternative embodiments, for game handle operation, the event code of the key event may include BTN_GAMEPAD, and BTN_GAMEPAD refers to a button on the game handle, which is a fixed function key on the game handle, and is usually used to control different operations in the game, such as starting the game, pausing, selecting, confirming, etc. BTN_GAMEPAD may have different names and positions on different game platforms and handles, but it is usually one of the important buttons used to control the game.

[0097] Exemplarily, the event codes of location events may include the following six types:

[0098] 1. ABS_MT_SLOT: Indicates the slot of the touch point, which can be used to distinguish the finger that controls the touch point. Each slot to which the touch point belongs is assigned a unique identifier. Through this event type, you can get the identifier of the slot where the current touch point is located.

[0099] 2. ABS_MT_TRACKING_ID: indicates the detection ID (tracking ID) of the touch point. Each touch point is assigned a unique detection ID. The detection ID of the current touch point can be obtained through the value of this event code. For example, a non-negative number indicates a touch, and ffffffff indicates that a touch ends, that is, the finger is lifted.

[0100] 3. ABS_MT_POSITION_X: Indicates the position of the touch point on the X-axis, which provides the horizontal position of the touch point relative to the upper left corner of the screen.

[0101] 4. ABS_MT_POSITION_Y: Indicates the position of the touch point on the Y axis, which provides the vertical position of the touch point relative to the upper left corner of the screen.

[0102] 5. ABS_MT_TOUCH_MAJOR: Indicates the main contact area of ​​the touch point, which provides the main area of ​​contact between the touch point and the screen.

[0103] 6. ABS_MT_TOUCH_MINOR: Indicates the secondary contact area of ​​the touch point, which provides the size of the secondary area of ​​contact between the touch point and the screen.

[0104] Among them, ABS_MT_SLOT is used to identify the slot where the touch point is located, and can be used to detect the status of multiple touch points simultaneously in a multi-touch device. ABS_MT_TRACKING_ID is used to identify the detection ID of the touch point, and can be used to detect the movement and continuous operation of the touch point. In a multi-touch device, each touch point will have a detection ID and an identifier of the slot where it is located. These two event codes can be used to accurately detect and process multiple touch points. ABS_MT_POSITION_X and ABS_MT_POSITION_Y provide the position information of the touch point on the X-axis and Y-axis of the screen. ABS_MT_TOUCH_MAJOR and ABS_MT_TOUCH_MINOR provide the contact area information of the touch point.

[0105] Exemplarily, the event code of the synchronization event may be SYN_REPORT, which represents the end of an event. Of course, in other alternative embodiments, the event code of the synchronization event may also include an event code representing the beginning of a time, which is not specifically limited in this application. Even if there is only the end of an event, it is possible to distinguish between events.

[0106] It is worth noting that the event types and event codes involved above are only examples of the present application and should not be construed as limiting the present application. For example, in other alternative embodiments, the event types and event codes may also be represented in other forms, or may include other event types, or may include other event codes.

[0107] For example, an example of the handle data is as follows:

[0108] / dev / input / event5:EV_KEY BTN_TL DOWN

[0109] / dev / input / event5:EV_SYN SYN_REPORT 00000000

[0110] / dev / input / event5:EV_KEY BTN_TL UP

[0111] / dev / input / event5:EV_SYN SYN_REPORT 00000000.

[0112] Among them, / dev / input / event5 is the reading path of the handle data, EV_KEY represents the key event, BTN_TL represents the upper left button (or trigger), DOWN represents pressing, and UP represents releasing. EV_SYN is a synchronization event used to synchronize the states of multiple events. 00000000 is a timestamp or counter. From the above data, we can know that the user pressed the upper left button (or trigger). For example, the upper left button can be the L1 button.

[0113] For example, another example of the handle data is as follows:

[0114] / dev / input / event5:EV_KEY BTN_GAMEPAD DOWN

[0115] / dev / input / event5:EV_SYN SYN_REPORT 00000000

[0116] / dev / input / event5:EV_KEY BTN_GAMEPAD UP

[0117] / dev / input / event5:EV_SYN SYN_REPORT 00000000.

[0118] Among them, / dev / input / event5 is the path of the input device, EV_KEY indicates a key event, BTN_GAMEPAD indicates a gamepad key, DOWN indicates press, and UP indicates release. EV_SYN is a synchronization event used to synchronize the states of multiple events. 00000000 is a timestamp or counter. From the above data, we can know that the user pressed a gamepad key. For example, the case may be key A.

[0119] S120: The data fusion device obtains a second event by converting the event code of the first event into an event code for identifying a touch event.

[0120] In other words, the data fusion device can obtain the second event by converting the event code of the first event; of course, the event type of the second event may be the same as or different from the event type of the first event, and the value of the second event may be the same as or different from the value of the first event, and the present application does not make any specific limitations on this.

[0121] In some embodiments, the S120 may include:

[0122] The data fusion device obtains the second event by replacing the event code of the first event with an event code for identifying a touch event while keeping the event type of the first event and the value of the first event unchanged.

[0123] In other words, the event type of the second event is the same as the event type of the first event, and the value of the second event is the same as the value of the first event.

[0124] That is to say, the event type of the second event is a key event type, the event code of the second event is used to identify the second event as a touch event, and the value of the second event is equal to the value of the first event. That is to say, the event types of the first event and the second event are the same, and the key operation that triggers the first event is the same as the touch operation that triggers the second event. For example, the event types of the first event and the second event are both key event types, and the key operation that triggers the first event and the touch operation that triggers the second event are both press operations.

[0125] For example, assume that the first event is as follows:

[0126] / dev / input / event5:EV_KEY BTN_GAMEPAD DOWN.

[0127] Among them, / dev / input / event5 is the path of the input device, EV_KEY represents the key event, BTN_GAMEPAD represents the game controller button, and DOWN represents the press operation.

[0128] Then the second event is:

[0129] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0130] Among them, / dev / input / event6 is the path of the input device, EV_ABS indicates an absolute event, ABS_MT_TRACKING_ID indicates the ID of the multi-touch, and DOWN indicates a press operation.

[0131] Similarly, suppose the first event is as follows:

[0132] / dev / input / event5:EV_KEY BTN_GAMEPAD UP.

[0133] Among them, / dev / input / event5 is the path of the input device, EV_KEY represents the key event, BTN_GAMEPAD represents the game controller button, and UP represents the release operation.

[0134] Then the second event is:

[0135] / dev / input / event6:EV_KEY BTN_TOUCH UP.

[0136] Among them, / dev / input / event6 is the path of the input device, EV_ABS indicates the absolute event, ABS_MT_TRACKING_ID indicates the ID of the multi-touch, and UP indicates the release operation.

[0137] S130: The data fusion device determines first touch data based on the second event.

[0138] Exemplarily, the first touch data is data generated by converting the handle data into touch data on a simulated touch screen. The first touch data includes the second event, that is, the first touch data is used to simulate finger operations on the touch screen, such as click, slide, zoom, etc.

[0139] In addition, the first touch data may also include information such as a touch point identifier, a touch point position, and a touch pressure.

[0140] S140, the data fusion device reads the second touch data from the screen input device node; the second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event.

[0141] Exemplarily, the second touch data is data generated by performing a touch operation on the touch screen. The second touch data includes the third event, that is, the first touch data is used to simulate the operation of a finger on the touch screen, such as click, slide, zoom, etc.

[0142] In addition, the second touch data may also include information such as a touch point identifier, a touch point position, and a touch pressure.

[0143] For example, an example of the third event may be as follows:

[0144] / dev / input / event1:EV_KEY BTN_TOUCH DOWN.

[0145] Exemplarily, another example of the third event may be as follows:

[0146] / dev / input / event1:EV_KEY BTN_TOOL_FINGER DOWN.

[0147] Exemplarily, an example of the second touch data is as follows:

[0148] / dev / input / event1:EV_ABS ABS_MT_SLOT 00000000

[0149] / dev / input / event1:EV_ABS ABS_MT_TRACKING_ID 00000016

[0150] / dev / input / event1:EV_KEY BTN_TOUCH DOWN

[0151] / dev / input / event1:EV_KEY BTN_TOOL_FINGER DOWN

[0152] / dev / input / event1:EV_ABS ABS_MT_POSITION_X 0000011a

[0153] / dev / input / event1:EV_ABS ABS_MT_POSITION_Y 00000475

[0154] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MAJOR 00000003

[0155] / dev / input / event1:EV_SYN SYN_REPORT 00000000

[0156] / dev / input / event1:EV_ABS ABS_MT_POSITION_X 0000011b

[0157] / dev / input / event1:EV_ABS ABS_MT_POSITION_Y 00000476

[0158] / dev / input / event1:EV_SYN SYN_REPORT 00000000

[0159] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MINOR 00000003

[0160] / dev / input / event1:EV_ABS ABS_MT_SLOT 00000001

[0161] / dev / input / event1:EV_ABS ABS_MT_TRACKING_ID 00000017

[0162] / dev / input / event1:EV_ABS ABS_MT_POSITION_X 00000324

[0163] / dev / input / event1:EV_ABS ABS_MT_POSITION_Y 00000165

[0164] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MAJOR 00000002

[0165] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MINOR 00000002

[0166] / dev / input / event1:EV_SYN SYN_REPORT 00000000

[0167] / dev / input / event1:EV ABS ABS_MT_POSITION_Y 00000166

[0168] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MAJOR 00000004

[0169] / dev / input / event1:EV_ABS ABS_MT_TOUCH_MINOR 00000003

[0170] / dev / input / event1:EV_SYN SYN_REPORT 00000000。

[0171] Among them, each / dev / input / event1 represents the path of the input device, EV_ABS represents an absolute event, EV_KEY represents a key event, BTN_TOUCH represents a touch event on the touch screen, and BTN_TOOL_FINGER represents a tool key using a finger. ABS_MT_SLOT represents a slot, ABS_MT_TRACKING_ID represents the ID of the multi-touch, ABS_MT_POSITION_X and ABS_MT_POSITION_Y represent the x and y coordinates of the touch, ABS_MT_TOUCH_MAJOR represents the major axis of the ellipse touched, and ABS_MT_TOUCH_MINOR represents the minor axis of the ellipse touched. EV_SYN is a synchronization event used to synchronize the states of multiple events. 00000000 is a timestamp or counter. From the above data, we can see that the first finger moved a little after pressing, and the second finger also moved after pressing.

[0172] S150: The data fusion device fuses the first touch data and the second touch data based on the reading order of the second event and the reading order of the third event to obtain fused data.

[0173] Exemplarily, the fused data is a result of fusing the first touch data and the second touch data. In this embodiment, by fusing the handle operation with the operation on the touch screen, a more flexible and diversified control method can be achieved. The fused data may include a combination of the buttons on the handle, the operation of the joystick, and the touch operation on the touch screen, as well as information such as the position and pressure corresponding to each operation.

[0174] S160, the data fusion device writes the fused data into the virtual input device node.

[0175] Exemplarily, a virtual input device node refers to an input device node simulated in a computer system, which is not a real physical device, but a virtual device for receiving input signals for human-computer interaction. In an operating system, a node can be assigned to each input device to receive input data sent by the device. These nodes can be hardware devices, such as keyboards, mice, touch pads, etc., or virtual devices, such as virtual keyboards, virtual mice, touch screen simulators, etc. The role of a virtual input device node in the system is similar to that of a real input device node. It can receive input data sent by an application and simulate corresponding user operations. Through the virtual input device node, an application can send simulated keyboard buttons, mouse clicks, touch events, etc. to the system to achieve functions such as automated operations, game control, and application testing. In short, a virtual input device node is a simulated input device interface. By writing the fused data into the virtual input device node, the user's input operation on the computer system can be simulated.

[0176] S170, the data fusion device reads the fusion data from the virtual input device node, and obtains and displays a rendered image based on the fusion data.

[0177] Exemplarily, the fused data may be used to determine screen data, which may be used to generate a rendered image. The screen data refers to image data on a handheld display screen, which includes pixel information for displaying a game, application, or operating system interface. The screen data may be represented as a two-dimensional array of images, where each element represents a color or brightness value of a pixel.

[0178] In this embodiment, the second event is obtained by converting the event code of the first event in the handle data, and then the first touch data is determined based on the second event; the second touch data is read from the screen input device node; based on the reading order of the second event and the reading order of the third event, the first touch data and the second touch data are fused to obtain fused data; the fused data is written into the virtual input device node; the fused data is read from the virtual input device node, and the rendered image is obtained and displayed based on the fused data, which can improve the compatibility of the handle and the touch screen and the use performance of the handle. For example, since the rendered image is obtained based on the fused data, when playing with the handle, even if the touch screen is accidentally touched, it will not cause the operation of the handle to fail, and will not cause errors in the game or application, thereby improving the use performance of the handle. For example, some games have some operations that are not convenient to use the handle or when the handle mapping points are insufficient, it is necessary to perform touch operations on the touch screen. In this embodiment, by fusing the first touch data and the second touch data, it can also meet the operation requirements of operating the handle and the touch screen at the same time, thereby improving the use performance of the handle.

[0179] In addition, the second event is obtained by converting the event code of the first event in the handle data, and then the first touch data is determined based on the second event. This is equivalent to converting the handle data into touch data from the system layer, which can ensure the compatibility between the first touch data and the touch data of the touch screen, thereby improving the success rate of data fusion.

[0180] In addition, the second touch data is read from the screen input device node; based on the reading order of the second event and the reading order of the third event, the first touch data and the second touch data are fused to obtain fused data, which can avoid the conflict between the first touch data and the second touch data in timing and improve the fusion effect of data fusion.

[0181] In some embodiments, before S130, the method 100 may further include:

[0182] Determine a mapping position of the key operation of the value identifier of the first event in the touch screen;

[0183] In any one of at least one preset direction, the distance between the mapping position and the reference point position in the touch screen is determined as the value of a fourth event; the event type of the fourth event is a position event type, the event code of the fourth event is used to identify the fourth event as an event for identifying the coordinate position in the any one preset direction, and the value of the fourth event is used to identify the mapping position in the any one preset direction.

[0184] Of course, the mapping position may also be referred to as a touch point position, that is, the value of the fourth event is used to identify the coordinate position of the touch point position in the preset direction;

[0185] Among them, the S130 may include:

[0186] The fourth event is spliced ​​after the second event to obtain the first touch data.

[0187] Exemplarily, assuming that the mapping position is (0x72, 0x693), and the preset direction includes the X direction and the Y direction, the fourth event includes:

[0188] / dev / input / event6:EV_ABS ABS_MT_POSITION_X 00000072

[0189] / dev / input / event6:EV_ABS ABS_MT_POSITION_Y 00000693.

[0190] In this case, if the second event is:

[0191] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0192] Then the first touch data may include:

[0193] / dev / input / event6:EV_KEY BTN_TOUCH DOWN

[0194] / dev / input / event6:EV_ABS ABS_MT_POSITION_X 00000072

[0195] / dev / input / event6:EV_ABS ABS_MT_POSITION_Y 00000693.

[0196] In some embodiments, when the key operation identified by the value of the first event is a press operation, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

[0197] Exemplarily, the at least one button may include all buttons of the handle.

[0198] Exemplarily, the correspondence between the at least one button and the at least one position in the touch screen may be pre-configured information. For example, the correspondence between the at least one button and the at least one position in the touch screen may be information obtained by performing a configuration operation on a mapping configuration interface. The mapping configuration interface may float above the game / application interface for easy viewing by the user. For example, the mapping configuration interface may be as follows: Figure 3 The interface shown.

[0199] In this embodiment, by introducing the corresponding relationship between the at least one button and at least one position in the touch screen, the mapping between the button and a screen coordinate point can be achieved. For example, by pressing the A button on the handle, the corresponding UI element position of the feature on the touch screen set by the user can be pressed, such as firing, thereby achieving the purpose of converting the first event into the second event and ensuring the accuracy of the second event.

[0200] In some embodiments, when the key identified by the event code of the first event is a multi-function key, the event code of the first event is also used to identify the operation mode of the multi-function key; when the key operation identified by the value of the first event is an operation to turn on the operation mode, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

[0201] Exemplarily, the multi-function button may be a joystick or a function button that can perform multiple operations.

[0202] It should be understood that the correspondence between the at least one button and the at least one position in the touch screen may be pre-configured information. For details, please refer to the above description. To avoid repetition, it will not be described here.

[0203] In some embodiments, the method 100 may further include:

[0204] In the case where the value of the first event is used to identify an operation for closing the operating mode, the mapping position is determined based on a correspondence between at least one mode and at least one moving direction and based on a moving direction in the at least one moving direction that corresponds to the operating mode.

[0205] Exemplarily, the at least one mode may include operations in up, down, left, and right directions.

[0206] Exemplarily, the at least one moving direction may include up, down, left, and right directions.

[0207] Exemplarily, the correspondence between the at least one mode and the at least one moving direction may be pre-configured information. For example, the correspondence between the at least one mode and the at least one moving direction may be information obtained through a configuration operation performed on a mapping configuration interface. The mapping configuration interface may float above the game / application interface for easy viewing by the user. For example, the mapping configuration interface may be as follows: Figure 3 The interface shown.

[0208] In this embodiment, by introducing a corresponding relationship between the at least one button and at least one position in the touch screen, the mapping between the joystick and the screen movement coordinates can be achieved. For example, by pushing the left joystick upwards, the character or object in the game can be controlled to move forward, that is, the joystick is mapped to screen press -> slide -> lift, thereby achieving the purpose of converting the first event into the second event and ensuring the accuracy of the second event.

[0209] In some embodiments, before S130, the method 100 may further include:

[0210] Allocating a touch point identifier for identifying a touch point of the second event;

[0211] The touch point is identified and determined as a value of a fifth event; the event type of the fifth event is a position event type, the event code of the fifth event is used to identify the fifth event as an event for identifying a touch point, and the value of the fifth event is used to identify the touch point of the second event;

[0212] Among them, the S130 may include:

[0213] The second event is spliced ​​after the fifth event to obtain the first touch data.

[0214] Exemplarily, the fifth event is:

[0215] / dev / input / event6: EV ABS ABS_MT_TRACKING_ID 00000007.

[0216] In this case, if the second event is:

[0217] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0218] Then the first touch data may include:

[0219] / dev / input / event6:EV ABS ABS_MT_TRACKING_ID 00000007

[0220] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0221] In some embodiments, before S130, the method 100 may further include:

[0222] Allocating a slot identifier for identifying the slot to which the touch point of the second event belongs;

[0223] The slot identifier is determined as a value of a sixth event; the event type of the sixth event is a position event type, the event code of the sixth event is used to identify the sixth event as an event for identifying a slot, and the value of the sixth event is used to identify the slot to which the touch point of the second event belongs;

[0224] Among them, the S130 may include:

[0225] The second event is spliced ​​after the sixth event to obtain the first touch data.

[0226] Exemplarily, when the touch point of the touch screen includes multiple touch points, the slot is identified and determined as the value of the sixth event.

[0227] Exemplarily, the sixth event is:

[0228] / dev / input / event6:EV_ABS ABS_MT_SLOT 00000000.

[0229] In this case, if the second event is:

[0230] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0231] Then the first touch data may include:

[0232] / dev / input / event6:EV_ABS ABS_MT_SLOT 00000000

[0233] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0234] In some embodiments, the second touch data includes a seventh event, the event type of the seventh event is a position event type, the event code of the seventh event is used to identify the seventh event as an event for identifying a slot, and the value of the seventh event is used to identify the slot to which the touch point of the third event belongs;

[0235] Before S150, the method 100 may further include:

[0236] When the identifier of the slot to which the touch point of the sixth event belongs is the same as the identifier of the slot to which the touch point of the seventh event belongs, different identifiers are reallocated to the slot to which the touch point of the sixth event belongs and the slot to which the touch point of the seventh event belongs based on the reading order of the second event and the reading order of the third event.

[0237] Exemplarily, if the reading order of the second event is earlier than the reading order of the third event, the identifier reallocated to the slot to which the touch point of the sixth event belongs is smaller than the identifier reallocated to the slot to which the touch point of the seventh event belongs; otherwise, the identifier reallocated by the data fusion device to the slot to which the touch point of the sixth event belongs is larger than the identifier reallocated to the slot to which the touch point of the seventh event belongs.

[0238] Exemplarily, if the reading order of the second event is earlier than the reading order of the third event, the identifier reallocated to the slot to which the touch point of the sixth event belongs is greater than the identifier reallocated to the slot to which the touch point of the seventh event belongs; otherwise, the identifier reallocated by the data fusion device to the slot to which the touch point of the sixth event belongs is less than the identifier reallocated to the slot to which the touch point of the seventh event belongs.

[0239] In some embodiments, the S130 may include:

[0240] A synchronization event is added after the second event to obtain the first touch data.

[0241] Exemplarily, the synchronization event is:

[0242] / dev / input / event6:EV_SYN SYN_REPORT 00000000.

[0243] In this case, if the second event is:

[0244] / dev / input / event6:EV_KEY BTN_TOUCH DOWN.

[0245] Then the first touch data may include:

[0246] / dev / input / event6:EV_KEY BTN_TOUCH DOWN

[0247] / dev / input / event6:EV_SYN SYN_REPORT 00000000.

[0248] It is worth noting that the first touch data may include at least one of the fourth event to the sixth event and the synchronization event mentioned above.

[0249] For example, suppose the handle data is as follows:

[0250] / dev / input / event5:EV_KEY BTN_GAMEPAD DOWN

[0251] / dev / input / event5:EV_SYN SYN_REPORT 00000000

[0252] / dev / input / event5:EV_KEY BTN_GAMEPAD UP

[0253] / dev / input / event5:EV_SYN SYN_REPORT 00000000.

[0254] Among them, / dev / input / event5 is the path of the input device, EV_KEY indicates a key event, BTN_GAMEPAD indicates a gamepad key, DOWN indicates a press operation, and UP indicates a release operation. EV_SYN is a synchronization event used to synchronize the states of multiple events. 00000000 is a timestamp or counter. From the above data, we can know that the user pressed a gamepad key. For example, the case can be key A.

[0255] In this embodiment, assuming that the mapped screen coordinates of the GAMEPAD button are configured as (0x72, 0x693), the handle data of the user pressing the GAMEPAD button can be converted into the first touch data according to the Linux input data standard. The first touch data may include:

[0256] / dev / input / event6:EV ABS ABS_MT_TRACKING_ID 00000007

[0257] / dev / input / event6:EV_KEY BTN_TOUCH DOWN

[0258] / dev / input / event6:EV_ABS ABS_MT_POSITION_X 00000072

[0259] / dev / input / event6:EV_ABS ABS_MT_POSITION_Y 00000693

[0260] / dev / input / event6:EV_SYN SYN_REPORT 00000000

[0261] / dev / input / event6:EV_ABS ABS_MT_TRACKING_ID ffffffff

[0262] / dev / input / event6:EV_KEY BTN_TOUCH UP

[0263] / dev / input / event6:EV_SYN SYN_REPORT 00000000.

[0264] Among them, / dev / input / event6 is the path of the input device, EV_ABS indicates an absolute event, ABS_MT_TRACKING_ID indicates the ID of the multi-touch, DOWN indicates a press operation, UP indicates a release operation, ABS_MT_POSITION_X and ABS_MT_POSITION_Y indicate the x and y coordinates of the touch. EV_SYN is a synchronization event used to synchronize the states of multiple events. 00000000 is a timestamp or counter.

[0265] In some embodiments, S110 may include:

[0266] In response to an opening operation performed on the mapping function control, the handle data is read from the handle input device node; the mapping function control is used to control opening or closing a function of the electronic device converting the data obtained from the handle input device node into touch data.

[0267] Exemplarily, in response to an opening operation performed on the mapping function control, the data fusion apparatus reads the handle data from the handle input device node when the handle data exists in the handle input device node.

[0268] For example, the opening operation may be a click operation performed on a "Open (or Close) Key Mapping" button on the quick setting panel. The quick setting panel may be as follows: Figure 2 The setting interface shown includes a "Turn on (or off) key mapping" button, a "Show mapped keys" button, and a "Configure key mapping" button.

[0269] In some embodiments, the method 100 may further include:

[0270] In response to the start operation, a reading path of data for image rendering is switched from the screen input device node to the virtual input device node.

[0271] In this embodiment, in response to the start operation performed on the mapping function control, the data fusion device switches the reading path of the data used for image rendering from the screen input device node to the virtual input device node, which can avoid the data fusion device reading the data used for image rendering from the screen input device node, thereby ensuring the accuracy of image rendering. In addition, the start operation not only triggers the data fusion device to read the handle data from the handle input device node, but also switches the reading path of the data used for image rendering, which can reduce the complexity of user operations during the data fusion process.

[0272] Figure 6 It is a schematic flowchart of the fusion data reading process provided in an embodiment of the present application.

[0273] like Figure 6 As shown, the reading process of the fused data may include:

[0274] S210, read handle data.

[0275] Specifically, the data fusion device detects the state of the handle input device node through a detector, and when the state of the handle input device node changes, the handle data is obtained by calling a callback function associated with the detector. For example, by creating epoll / poll to detect the file descriptor (fd) of the handle input device node (for example, by inputting the name of the device node to select the handle input device node) and setting a callback, when the user operates the handle, the state of the handle input device node changes, and the handle data is obtained by calling the set callback function.

[0276] S220: Read second touch data

[0277] Specifically, the data fusion device detects the state of the screen input device node through a detector, and when the state of the screen input device node changes, the screen data is obtained by calling a callback function associated with the detector. For example, by creating epoll / poll to detect the file descriptor (fd) of the screen input device node (for example, by inputting the name of the device node to select the screen input device node) and setting a callback, when the user operates the screen, the state of the screen input device node changes, and the screen data is obtained by calling the set callback function.

[0278] S230: Mapping process, which includes a data conversion process and a data fusion process.

[0279] For the data conversion process, the handle input data complies with the Linux input data standard, which is divided into button lift, button press, joystick position on the direction axis (including X-axis and Y-axis), and trigger stroke size. This data needs to be mapped to the screen as touch lift, touch press, touch move, etc. The touch data also needs to comply with the Linux input data standard and can be converted according to the standard.

[0280] After the handle data is converted to obtain the first touch data, a data fusion process is performed on the first touch data and the second touch data read from the screen input device node.

[0281] For the data fusion process, multi-touch data has a concept of slots, which marks the data of which finger is pressed and moved. Only after the first touch data and the second touch data are mastered at the same time, can the slot identification be correctly assigned to each touch data, otherwise the system will abandon the data or an error will occur due to the conflict of slot identification. Since this application reads the data of two device nodes at the same time, the identification of the slot of each touch data can be obtained, and the slot of each touch data can be reallocated or set with the correct identification in a queue in a first-come-first-served manner.

[0282] Figure 7 This is an example of the data fusion process provided in the embodiments of the present application.

[0283] like Figure 7 As shown, it is assumed that the data fusion device has read 4 second touch data, which are data A (the slot identifier is 0), data B (the slot identifier is 1), data C (the slot identifier is 2), and data D (the slot identifier is 0), wherein the slot identifier of data A and the slot identifier of data D are the same, both are 0, which means that the two data are generated by the same finger, and the user presses the finger (generating data A) and slides on the screen (generating data D). At the same time, the data fusion device reads 3 handle data and converts them into 3 first touch data, which are data E (the slot identifier is 0), data F (the slot identifier is 1), and data G (the slot identifier is 0), wherein the slot identifier of data E and the slot identifier of data G are the same, which is 0, which means that the two data are generated by operating the same handle button, with a sliding operation, which may be the left joystick or the right joystick.

[0284] Based on this, the data fusion device can put the data into a queue according to the reading order of each data, and the queue becomes data E, data A, data B, data F, data C, data D, and data G. In addition, the slots of each data need to be reassigned with identifiers. For example, the slots of each data can be reassigned with identifiers in order, that is, the identifier of the slot of data E is 0, the identifier of the slot of data A is 1, the identifier of the slot of data B is 2, the identifier of the slot of data F is 3, and the identifier of the slot of data C is 4. In addition, in order to maintain the relationship between the identifiers of the slots of each data, that is, data A and data D are the same finger event, so the identifier of the slot of data D also needs to be kept the same as the identifier of the slot of data A, that is, the identifier of the slot of data D is 1, and data E and data G are the same finger event, the identifier of the slot of data G also needs to be kept the same as the identifier of the slot of data E, that is, the identifier of the slot of data G is 0.

[0285] S240: Write fusion data.

[0286] Specifically, the virtual input device node is a virtual input device supported by the Android system without the need for real hardware. The system will treat it as a real input device, read the input and distribute it. In this embodiment, the fused data uses the virtual input device node to perform input operations. For example, you can first create a virtual input device node, and then write the fused data to the virtual input device node. For the creation of the virtual input device node, you can open the " / dev / uinput" file directory, create a device node with a structure of uinput_user_dev, and configure it as a touch device, and then create it through the IOCtl interface. For the writing process of the fused data, you can directly write the data to the virtual input device node file, and the Android system will be responsible for processing it later.

[0287] It is worth noting that in addition to the above steps, you also need to disable the system screen input data, otherwise the system will read two touch data at the same time. For example, when the key mapping function is turned on, call the corresponding system interface to disable the screen input.

[0288] In this embodiment, by introducing a mapping process and designing it as a system process, it is responsible for reading the handle data from the handle input device node and then converting it into the first touch data, reading the second screen data from the screen input device node, and writing a virtual input device node after fusing the two. In addition, when the key mapping function is turned on, the screen input device node data is disabled at the system level, thereby achieving compatibility between key mapping and screen data.

[0289] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the embodiments mentioned above. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the specific embodiments mentioned above can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0290] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the processes involved above does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0291] The method provided in the embodiment of the present application is described above, and the device provided in the embodiment of the present application is described below.

[0292] Figure 8 It is a schematic block diagram of a data fusion device 300 provided in an embodiment of the present application.

[0293] like Figure 8 As shown, the data fusion device 300 may include:

[0294] The first reading unit 310 is used to read the handle data from the handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used to identify the key that triggers the first event, and the value of the first event is used to identify the key operation that triggers the first event;

[0295] A conversion unit 320, configured to obtain a second event by converting the event code of the first event into an event code for identifying a touch event;

[0296] A determining unit 330, configured to determine first touch data based on the second event;

[0297] The second reading unit 340 is used to read second touch data from the screen input device node; the second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event;

[0298] A fusion unit 350, configured to fuse the first touch data and the second touch data based on a reading order of the second event and a reading order of the third event to obtain fused data;

[0299] A writing unit 360, used for writing the fused data into the virtual input device node;

[0300] The rendering unit 370 is used to read the fused data from the virtual input device node, and obtain and display a rendered image based on the fused data.

[0301] In some embodiments, before determining the first touch data based on the second event, the determining unit 330 is further configured to:

[0302] Determine a mapping position of the key operation of the value identifier of the first event in the touch screen;

[0303] In any one of at least one preset direction, the distance between the mapping position and the reference point position in the touch screen is determined as the value of a fourth event; the event type of the fourth event is a position event type, the event code of the fourth event is used to identify the fourth event as an event for identifying a coordinate position in the any one preset direction, and the value of the fourth event is used to identify the coordinate of the mapping position in the any one preset direction;

[0304] The determining unit 330 is specifically used for:

[0305] The fourth event is spliced ​​after the second event to obtain the first touch data.

[0306] In some embodiments, the determining unit 330 is specifically configured to:

[0307] When the key operation identified by the value of the first event is a pressing operation, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

[0308] In some embodiments, when the key identified by the event code of the first event is a multi-function key, the event code of the first event is also used to identify the operation mode of the multi-function key;

[0309] The determining unit 330 is specifically used for:

[0310] When the key operation identified by the value of the first event is an operation to start the operation mode, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

[0311] In some embodiments, the determining unit 330 is specifically configured to:

[0312] In the case where the value of the first event is used to identify an operation for closing the operating mode, the mapping position is determined based on a correspondence between at least one mode and at least one moving direction and based on a moving direction in the at least one moving direction that corresponds to the operating mode.

[0313] In some embodiments, before determining the first touch data based on the second event, the determining unit 330 is further configured to:

[0314] Allocating a touch point identifier for identifying a touch point of the second event;

[0315] The touch point is identified and determined as a value of a fifth event; the event type of the fifth event is a position event type, the event code of the fifth event is used to identify the fifth event as an event for identifying a touch point, and the value of the fifth event is used to identify the touch point of the second event;

[0316] The determining unit 330 is specifically used for:

[0317] The second event is spliced ​​after the fifth event to obtain the first touch data.

[0318] In some embodiments, before determining the first touch data based on the second event, the determining unit 330 is further configured to:

[0319] Allocating a slot identifier for identifying the slot to which the touch point of the second event belongs;

[0320] The slot identifier is determined as a value of a sixth event; the event type of the sixth event is a position event type, the event code of the sixth event is used to identify the sixth event as an event for identifying a slot, and the value of the sixth event is used to identify the slot to which the touch point of the second event belongs;

[0321] The determining unit 330 is specifically used for:

[0322] The second event is spliced ​​after the sixth event to obtain the first touch data.

[0323] In some embodiments, the second touch data includes a seventh event, the event type of the seventh event is a position event type, the event code of the seventh event is used to identify the seventh event as an event for identifying a slot, and the value of the seventh event is used to identify the slot to which the touch point of the third event belongs;

[0324] Wherein, before fusing the first touch data and the second touch data based on the reading order of the second event and the reading order of the third event, the fusing unit 350 is further used to:

[0325] When the identifier of the slot to which the touch point of the sixth event belongs is the same as the identifier of the slot to which the touch point of the seventh event belongs, different identifiers are reallocated to the slot to which the touch point of the sixth event belongs and the slot to which the touch point of the seventh event belongs based on the reading order of the second event and the reading order of the third event.

[0326] In some embodiments, the determining unit 330 is specifically configured to:

[0327] A synchronization event is added after the second event to obtain the first touch data.

[0328] In some embodiments, the first reading unit 310 is specifically used for:

[0329] In response to an opening operation performed on the mapping function control, the handle data is read from the handle input device node; the mapping function control is used to control opening or closing a function of the electronic device converting the data obtained from the handle input device node into touch data.

[0330] In some embodiments, the rendering unit 370 is further configured to:

[0331] In response to the start operation, a reading path of data for image rendering is switched from the screen input device node to the virtual input device node.

[0332] In some embodiments, the conversion unit 320 is specifically used to:

[0333] While keeping the event type of the first event and the value of the first event unchanged, the second event is obtained by replacing the event code of the first event with an event code for identifying a touch event.

[0334] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the data fusion device 300 may correspond to the corresponding subject in the method 100 of the embodiment of the present application, and the various units in the data fusion device 300 are respectively for implementing the corresponding processes in the method 100, and for the sake of brevity, no further description is given here.

[0335] It should also be understood that the various units in the data fusion device 300 involved in the embodiment of the present application are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit, and even these functions can be implemented with the assistance of one or more other units. For example, part or all of the data fusion device 300 is merged into one or more other units. For another example, a certain unit (or units) in the data fusion device 300 can also be split into multiple smaller units in function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiment of the present application. For another example, the data fusion device 300 can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0336] According to another embodiment of the present application, the data fusion device 300 involved in the embodiment of the present application can be constructed by running a computer program (including program code) capable of executing each step involved in the corresponding method on a general computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM), so as to realize the method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device through a computer-readable storage medium, and run therein to realize the corresponding method of the embodiment of the present application. In other words, the units involved above can be realized in the form of hardware, or can be realized by instructions in the form of software, or can be realized in the form of a combination of software and hardware. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software in the decoding processor to execute. Optionally, the software can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, and a register. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the method embodiment mentioned above in combination with its hardware.

[0337] Fig. 9 It is a schematic block diagram of an electronic device provided in an embodiment of the present application.

[0338] like Fig. 9 As shown, the electronic device 400 at least includes a processor 410 and a computer-readable storage medium 420. The processor 410 and the computer-readable storage medium 420 may be connected via a bus or other means. The computer-readable storage medium 420 is used to store a computer program 421, which includes computer instructions, and the processor 410 is used to execute the computer instructions stored in the computer-readable storage medium 420. The processor 410 is the computing core and control core of the electronic device 400, which is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0339] As an example, the processor 410 may also be referred to as a central processing unit (CPU). The processor 410 may include, but is not limited to, a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete component gates or transistor logic devices, discrete hardware components, and the like.

[0340] As an example, the computer-readable storage medium 420 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 410. Specifically, the computer-readable storage medium 420 includes, but is not limited to: a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0341] like Fig. 9 As shown, the electronic device 400 may further include a transceiver 430 .

[0342] The processor 410 may control the transceiver 430 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0343] It should be understood that the various components in the electronic device 400 are connected via a bus system, wherein the bus system includes a power bus, a control bus, and a status signal bus in addition to a data bus. It is worth noting that the electronic device 400 can be any electronic device with data processing capabilities; the computer-readable storage medium 420 stores a first computer instruction; the processor 410 loads and executes the first computer instruction stored in the computer-readable storage medium 420 to implement Figure 1 The corresponding steps in the method embodiment shown; in a specific implementation, the first computer instruction in the computer-readable storage medium 420 is loaded by the processor 410 and executes the corresponding steps. To avoid repetition, they are not repeated here.

[0344] According to another aspect of the present application, an embodiment of the present application provides a chip. The chip may be an integrated circuit chip having signal processing capabilities, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The chip may also be referred to as a system-on-chip, a system-on-chip, a chip system, or a system-on-chip chip, etc. The chip may be applied to various electronic devices capable of installing the chip, so that the device equipped with the chip may execute the corresponding steps in the methods or logic block diagrams disclosed in the embodiments of the present application. For example, the chip may be suitable for implementing one or more computer instructions, specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0345] According to another aspect of the present application, an embodiment of the present application provides a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device of a computer for storing programs and data. It is understandable that the computer-readable storage medium herein may include both a built-in storage medium in a computer and, of course, an extended storage medium supported by a computer. The computer-readable storage medium provides a storage space, which stores an operating system of an electronic device. The storage space stores computer instructions suitable for being loaded and executed by a processor, and when the computer instructions are read and executed by a processor of a computer device, the computer device executes the corresponding steps in each method or logic block diagram disclosed in the embodiment of the present application.

[0346] According to another aspect of the present application, an embodiment of the present application provides a computer program product or a computer program. The computer program product or computer program includes a computer instruction, which is stored in a computer-readable storage medium. The processor of a computer device reads the computer instruction from a computer-readable storage medium, and the processor executes the computer instruction so that the computer device executes the corresponding steps in each method disclosed in the embodiment of the present application or the logic block diagram. In other words, when the solution provided by the present application is implemented using software, it can be implemented in whole or in part in the form of a computer program product or a computer program. The computer program product or computer program includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the function of the embodiment of the present application is implemented.

[0347] It is worth noting that the computer involved in the present application can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions involved in the present application can be stored in a computer-readable storage medium, or can be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0348] Those of ordinary skill in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. In other words, professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0349] Finally, it should be noted that the above content is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims. For example, the various specific technical features described in the above specific implementation methods can be combined in any suitable manner without contradiction. For another example, the various implementation methods of the present application can also be arbitrarily combined, as long as they do not violate the basic idea of ​​the present application, they should also be regarded as the content disclosed in the present application.

Claims

1. A data fusion method, characterized in that: include: Reading handle data from a handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used to identify a key that triggers the first event, and the value of the first event is used to identify the key operation that triggers the first event; Obtaining a second event by converting the event code of the first event into an event code for identifying a touch event; determining first touch data based on the second event; Read the second touch data from the screen input device node; The second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event; Based on a reading order of the second event and a reading order of the third event, the first touch data and the second touch data are merged to obtain fused data; Writing the fused data into a virtual input device node; The fused data is read from the virtual input device node, and a rendered image is obtained and displayed based on the fused data.

2. The method according to claim 1, characterized in that Before determining the first touch data based on the second event, the method further includes: Determine a mapping position of the key operation of the value identifier of the first event on the touch screen; In any one of at least one preset direction, the distance between the mapping position and the reference point position in the touch screen is determined as the value of a fourth event; the event type of the fourth event is a position event type, the event code of the fourth event is used to identify the fourth event as an event for identifying a coordinate position in any one of the preset directions, and the value of the fourth event is used to identify the coordinate position of the mapping position in any one of the preset directions; Wherein, determining the first touch data based on the second event includes: The fourth event is spliced ​​after the second event to obtain the first touch data.

3. The method according to claim 2, characterized in that The determining of a mapping position of the key operation of the value identifier of the first event on the touch screen includes: In a case where the key operation identified by the value of the first event is a pressing operation, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

4. The method according to claim 3, characterized in that When the key identified by the event code of the first event is a multi-function key, the event code of the first event is also used to identify the operation mode of the multi-function key; The step of determining a mapping position of the key operation of the value identifier of the first event on the touch screen includes: In a case where the key operation identified by the value of the first event is an operation for starting the operation mode, based on the correspondence between at least one key and at least one position in the touch screen, the position in the at least one position corresponding to the key identified by the event code of the first event is determined as the mapping position.

5. The method according to claim 4, characterized in that The method further comprises: In the case where the value of the first event is used to identify an operation for closing the operating mode, the mapping position is determined based on a correspondence between at least one mode and at least one moving direction and based on a moving direction in the at least one moving direction that corresponds to the operating mode.

6. The method according to claim 1, characterized in that Before determining the first touch data based on the second event, the method further includes: assigning a touch point identifier for identifying a touch point of the second event; The touch point identifier is determined as a value of a fifth event; the event type of the fifth event is a position event type, the event code of the fifth event is used to identify the fifth event as an event for identifying a touch point, and the value of the fifth event is used to identify the touch point of the second event; Wherein, determining the first touch data based on the second event includes: A second event is spliced ​​after the fifth event to obtain the first touch data.

7. The method according to claim 1, characterized in that Before determining the first touch data based on the second event, the method further includes: Allocating a slot identifier for identifying the slot to which the touch point of the second event belongs; The slot identifier is determined as a value of a sixth event; the event type of the sixth event is a position event type, the event code of the sixth event is used to identify the sixth event as an event for identifying a slot, and the value of the sixth event is used to identify the slot to which the touch point of the second event belongs; Wherein, determining the first touch data based on the second event includes: A second event is spliced ​​after the sixth event to obtain the first touch data.

8. The method according to claim 7, characterized in that The second touch data includes a seventh event, the event type of the seventh event is a position event type, the event code of the seventh event is used to identify the seventh event as an event for identifying a slot, and the value of the seventh event is used to identify the slot to which the touch point of the third event belongs; Before fusing the first touch data and the second touch data based on the reading order of the second event and the reading order of the third event, the method further includes: When the identifier of the slot to which the touch point of the sixth event belongs is the same as the identifier of the slot to which the touch point of the seventh event belongs, different identifiers are reallocated to the slot to which the touch point of the sixth event belongs and the slot to which the touch point of the seventh event belongs based on the reading order of the second event and the reading order of the third event.

9. The method according to claim 1, characterized in that: The determining the first touch data based on the second event includes: A synchronization event is added after the second event to obtain the first touch data.

10. The method according to claim 1, characterized in that The step of reading the handle data from the handle input device node includes: In response to an activation operation performed on a mapping function control, the handle data is read from the handle input device node; the mapping function control is used to control activation or deactivation of a function in which the electronic device converts the data obtained from the handle input device node into touch data.

11. The method according to claim 10, characterized in that The method further comprises: In response to the start operation, a reading path of data for image rendering is switched from the screen input device node to the virtual input device node.

12. The method according to claim 1, characterized in that The step of converting the event code of the first event into an event code for identifying a touch event to obtain the second event includes: While keeping the event type of the first event and the value of the first event unchanged, the second event is obtained by replacing the event code of the first event with an event code for identifying a touch event.

13. A data fusion device, characterized in that: include: A first reading unit, used for reading handle data from a handle input device node; the handle data includes a first event, the event type of the first event is a key event type, the event code of the first event is used for identifying a key that triggers the first event, and the value of the first event is used for identifying the key operation that triggers the first event; a conversion unit, configured to obtain a second event by converting an event code of the first event into an event code for identifying a touch event; a determining unit, configured to determine first touch data based on the second event; A second reading unit, used for reading second touch data from the screen input device node; The second touch data includes a third event, the event type of the third event is a key event type, the event code of the third event is used to identify the third event as a touch event, and the value of the third event is used to identify the touch operation that triggers the third event; a fusion unit, configured to fuse the first touch data and the second touch data based on a reading order of the second event and a reading order of the third event to obtain fused data; A writing unit, used for writing the fused data into a virtual input device node; A rendering unit is used to read the fused data from the virtual input device node, and obtain and display a rendered image based on the fused data.

14. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 12.

Images