Electronic device, server, and method of operation thereof for controlled experiment

Abstract

An electronic device according to one embodiment may comprise: a communication circuit; at least one processor; and a memory that stores instructions. The instructions according to one embodiment may be configured to, when executed by the at least one processor, instruct an electronic device to transmit, to a server, user characteristic data collected while a plurality of applications are executed in the electronic device, and, upon receiving an experimental model determined on the basis of at least a portion of the user characteristic data from a server, update a first application supporting a control experiment among the plurality of applications using the received experimental model, and transmit, to the server, user usage data collected while the updated first application is executed.

Description

Electronic device, server and method of operation thereof for controlled experiments

[0001] The present disclosure relates to an electronic device, a server and a method of operating the same for a control experiment.

[0002] AB testing is an experiment that compares Plan A, consisting of existing elements, with Plan B, which modifies certain elements, to determine which performs better. Plan A is displayed on some users' electronic devices, while Plan B is displayed on others, and the performance is compared. This is commonly conducted on websites, emails, and advertisements for internet services.

[0003] AB testing is the most practical and immediate way to create a culture where corporate decision-making is data-driven. AB testing platforms provide a way to validate hypotheses with real users and measurable results, replacing internal opinions and debates with real behavioral data.

[0004] AB testing divides users into several groups and exposes them to different buttons and UIs on the website to measure subscription or donation success rates to determine the optimal solution.

[0005] Beyond UI, AB testing is also possible to compare and analyze the effectiveness of models (algorithms). Various models (algorithms) are available for content recommendation for movie and music services. For example, a recommendation model (algorithm) for a music service can be used, as shown below.

[0006] - Popularity (Weekly / Real-time chart)

[0007] - Music by the user's favorite singer

[0008] - Music of the user's preferred genre

[0009] - Music that suits today's weather

[0010] Determining which model (algorithm) is most effective is a difficult question. Therefore, having actual online testing data allows for rational, data-driven decisions. A / B testing allows for the application of different models (algorithms) to experimental and control groups, and comparisons of retention rates and usage times can be used to determine the optimal solution.

[0011] The electronic device and server for the control experiment can provide an optimal model that can update the application to provide an application that suits the user characteristics of the electronic device.

[0012] The electronic device and server for the controlled experiment can provide multiple experimental groups that are statistically similar.

[0013] The electronic device and server for the controlled experiment can determine an optimal model for updating the application based on the experimental results using the plurality of experimental groups to provide an application tailored to the user characteristics of the electronic device.

[0014] According to one embodiment, an electronic device may include a communication circuit, at least one processor, and a memory storing instructions. According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to transmit user characteristic data collected while a plurality of applications are executed on the electronic device to a server. According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to receive an experimental model determined based on at least a portion of the user characteristic data from a server, and update a first application among the plurality of applications using the received experimental model. According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to transmit user usage data collected while the updated first application is executed to the server.

[0015] According to one embodiment, a server may include a communication circuit, at least one processor, and a memory storing instructions. The instructions according to one embodiment may be configured to cause the electronic devices to determine a plurality of experimental groups among the plurality of electronic devices based on user characteristic data received from the plurality of electronic devices when executed by the at least one processor. The instructions according to one embodiment may be configured to cause the electronic devices to determine a plurality of experimental models associated with an application to be set for each of the plurality of experimental groups when executed by the at least one processor. The instructions according to one embodiment may be configured to cause the electronic devices to transmit a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups when executed by the at least one processor. The commands according to one embodiment may be configured, when executed by the at least one processor, to cause the electronic devices to transmit a second experimental model from among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group from among the plurality of experimental groups. The commands according to one embodiment may be configured, when executed by the at least one processor, to cause the electronic devices to determine an experimental model for the plurality of experimental groups from among the plurality of experimental models based on user usability data received from the plurality of experimental groups.

[0016] In one embodiment, a method for operating an electronic device may include transmitting user characteristic data collected while a plurality of applications are running on the electronic device to a server. In one embodiment, the method may include receiving an experimental model determined based on at least a portion of the user characteristic data from the server, and updating a first application among the plurality of applications using the received experimental model. In one embodiment, the method may include transmitting user usage data collected while the updated first application is running to the server.

[0017] According to one embodiment, a method operating on a server may include an operation of determining a plurality of experimental groups among a plurality of electronic devices based on user characteristic data received from the plurality of electronic devices. According to one embodiment, the method may include an operation of determining a plurality of experimental models related to an application to be set for each of the plurality of experimental groups. According to one embodiment, the method may include an operation of transmitting a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups. According to one embodiment, the method may include an operation of transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups. According to one embodiment, the method may include an operation of determining an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user's usability data received from the plurality of experimental groups.

[0018] In one embodiment, a non-volatile storage medium storing commands, wherein the commands are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include transmitting user characteristic data collected while a plurality of applications are executed on the electronic device to a server. In one embodiment, the at least one operation may include receiving an experiment model determined based on at least a portion of the user characteristic data from the server, and updating a first application among the plurality of applications using the received experiment model. In one embodiment, the at least one operation may include transmitting user usage data collected while the updated first application is executed to the server.

[0019] In one embodiment, a non-volatile storage medium storing commands is provided, wherein the commands, when executed by an electronic device, are configured to cause the electronic device to perform at least one operation, wherein the at least one operation may include an operation of determining a plurality of experimental groups among the plurality of electronic devices based on user characteristic data received from the plurality of electronic devices. The at least one operation according to one embodiment may include an operation of determining a plurality of experimental models related to an application to be set for each of the plurality of experimental groups. The at least one operation according to one embodiment may include an operation of transmitting a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups. The at least one operation according to one embodiment may include an operation of transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups. The at least one operation according to one embodiment may include an operation of determining an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user's usability data received from the plurality of experimental groups.

[0020] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0021] Figure 2 is a block diagram of a server for a control experiment according to one embodiment.

[0022] Figure 3 is a block diagram of an electronic device according to one embodiment.

[0023] FIG. 4 is a diagram illustrating an operation of determining multiple experimental groups based on user characteristic data in a server according to one embodiment.

[0024] FIGS. 5A, 5B, and 5C are diagrams illustrating an operation of determining multiple experimental groups based on user characteristic data by a server and an electronic device according to one embodiment.

[0025] FIGS. 6A and 6B are diagrams illustrating an operation of multiple experimental groups executing an application in an electronic device according to one embodiment.

[0026] FIGS. 7A and 7B are diagrams illustrating experimental results through multiple experimental groups in a server according to one embodiment.

[0027] FIGS. 8A, 8B, and 8C are flowcharts illustrating operations for executing a control experiment by a server and an electronic device according to one embodiment.

[0028] Figure 9 is a flowchart illustrating an operation of providing a model from a server according to one embodiment.

[0029] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to an embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to an embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0030] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0031] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0032] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0033] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0034] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0035] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0036] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0037] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0038] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0039] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0040] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0042] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0043] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0044] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0046] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0047] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0048] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0049] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0050] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0051] Figure 2 is a block diagram of a server according to one embodiment.

[0052] Referring to the above drawing 2, the server (210) may include a first processor (220), a control experiment unit (230), a first memory (250), and a first communication circuit (290). According to one embodiment, the first processor (220) may control the overall operation of the server.

[0053] According to one embodiment, the first processor (220) can control the control experiment unit (230) to execute a control experiment, and can execute the same function as the control experiment unit (230) or include the control experiment unit (230).

[0054] According to an embodiment, a control experiment unit (230) may determine a plurality of experimental groups among a plurality of electronic devices (310) based on user characteristic data received from the plurality of electronic devices, and may determine a plurality of experimental models related to applications to be set for each of the plurality of groups. According to an embodiment, the control experiment unit (230) may transmit the plurality of experimental models to the plurality of groups, and may determine an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user usability data received from the plurality of groups.

[0055] According to one embodiment, the control experiment unit (230) may include a group management unit (201), a group decision unit (203), a model provision unit (205), and an experiment analysis unit (207).

[0056] According to one embodiment, the group management unit (201) receives user characteristic data received from a plurality of electronic devices (310) periodically or at a specified time, stores the user characteristic data, and manages the plurality of electronic devices (310) by classifying them into a plurality of groups based on the user characteristic data.

[0057] The user characteristic data according to one embodiment may represent data collected while multiple applications are being used on each of the multiple electronic devices.

[0058] The user's characteristic data according to one embodiment may include first characteristic data collected while a first application supporting a control experiment is executed (used) on each of the plurality of electronic devices, and second characteristic data converted into keywords from a plurality of signals detected while a plurality of applications are executed on each of the plurality of electronic devices.

[0059] According to one embodiment, the group management unit (201) can classify and manage the plurality of electronic devices (310) into a plurality of groups based on first characteristic data collected while a first application supporting a control experiment among the plurality of applications included in the user's characteristic data is executed (used).

[0060] The first characteristic data according to one embodiment may include at least one of the usage time, the number of exposures per function, the frequency of use, the number of inputs per function, or the amount of input data for the first application supporting the control experiment.

[0061] For example, the group management unit (201) can compare the usage time of the first application with group conditions and classify and manage the plurality of electronic devices (310) into a plurality of groups based on the usage time of the first application.

[0062] According to one embodiment, the group management unit (201) can classify and manage the plurality of electronic devices (310) into a plurality of groups based on second characteristic data that converts a plurality of signals (information) detected while a plurality of applications are executed in each of the plurality of electronic devices (310) included in the characteristic data of the user into keywords.

[0063] The second characteristic data according to one embodiment may include keywords such as work, study, family, etc., and scores for the keywords.

[0064] For example, when the group management unit (201) receives second characteristic data including “task = 1” from a first electronic device (301) among the plurality of electronic devices (310), it can include the first electronic device in a group that includes “task = 1” based on a group condition.

[0065] In one embodiment, the group management unit (201) may receive user usage data collected while a first application updated with an experimental model is executed (used) from a plurality of electronic devices included in a plurality of experimental groups during a control experimental period, and may store the user usage data for each of the plurality of experimental groups.

[0066] The user's usability data according to one embodiment may include the duration of use, frequency of use, and / or amount of input data of the first application updated with the experimental model.

[0067] According to one embodiment, the group management unit (201) may provide a plurality of group information classified based on user characteristic data based on a request from the group determination unit (203).

[0068] According to one embodiment, the group management unit (201) may receive and store information on the control experiment period from the group determination unit (203) and information related to a plurality of experimental groups determined by the group determination unit (203).

[0069] According to one embodiment, the group management unit (201) may provide usability data stored for each of the plurality of experimental groups based on a request from the experimental analysis unit (209).

[0070] A group determination unit (203) according to one embodiment can determine a plurality of experimental groups for a control experiment among a plurality of groups.

[0071] The group determination unit (203) according to one embodiment can randomly determine multiple experimental groups for a control experiment among multiple groups.

[0072] According to one embodiment, the group determination unit (203) may determine a first group and a second group having user characteristic data that meet the experimental conditions among the plurality of groups based on the experimental conditions input by the experimenter, determine a first experimental group from the first group, and determine a second experimental group from the second group.

[0073] The group determination unit (203) according to one embodiment may determine a first group and a second group having user characteristic data that meet the experimental conditions among the plurality of groups based on the experimental conditions input by the experimenter, and may determine a first experimental group and a second experimental group from the first group, and / or determine a first experimental group and a second experimental group from the second group.

[0074] According to an embodiment, the group determination unit (203) may determine a plurality of experimental models related to a first application to be set in each of the plurality of experimental groups. The group determination unit (203) may determine a first experimental model among the plurality of experimental models in a first experimental group including a plurality of first electronic devices among the plurality of electronic devices as an experimental model related to the first application. The group determination unit (203) may determine a second experimental model among the plurality of experimental models in a second experimental group including a plurality of second electronic devices among the plurality of electronic devices as an experimental model related to the first application.

[0075] The group determination unit (203) according to an embodiment may receive information about the plurality of groups from the group management unit (201), and set a model capable of updating the first application based on the type of the first application corresponding to the characteristic data of the user classified into each of the plurality of groups. The group determination unit (203) according to an embodiment may differently change at least one of the properties, setting variables, functions, UI elements, statistical models, or algorithms included in the plurality of experimental models in order to update the first application under different conditions. The group determination unit (203) according to an embodiment may set the plurality of experimental groups to the optimized experimental model when receiving information about an optimized experimental model for the plurality of experimental groups from the experimental analysis unit (207) after the end of the experimental period.

[0076] According to one embodiment, the group decision unit (203) may request information on a plurality of groups from the group management unit (201) and set the plurality of groups to the optimized experimental model.

[0077] According to one embodiment, the group determination unit (203) may transmit model information set for a group including the electronic device to the model provision unit (205) based on the identification information of the electronic device received from the model provision unit (205) at the request of the model provision unit (205).

[0078] A model providing unit (205) according to one embodiment can transmit a plurality of experimental models to a plurality of electronic devices included in a plurality of experimental groups.

[0079] According to an embodiment, the model providing unit (205), when confirming receipt of an update request from a first electronic device (301) among the plurality of electronic devices, requests model information of the first electronic device together with identification information of the first electronic device to the group determining unit (203), and when receiving first experiment model information set for a first experiment group including the first electronic device (301) among the plurality of experiment groups from the group determining unit (201), transmits the first experiment model to the first electronic device (301).

[0080] According to an embodiment, the model providing unit (205), when confirming receipt of an update request from a first electronic device (301) among the plurality of electronic devices, requests model information of the first electronic device together with identification information of the first electronic device to the group determining unit (203), and when receiving model information set for a third group including the first electronic device (301) among the plurality of groups not included in the plurality of experimental groups from the group determining unit (201), the model set for the third group can be transmitted to the first electronic device (301). According to an embodiment, when receiving a basic model from the group determining unit (201) because a model for the third group is not set, the model providing unit (205) can transmit the basic model to the first electronic device (301).

[0081] According to an embodiment, the model providing unit (205) may, upon confirming receipt of an update request from a first electronic device (301) among the plurality of electronic devices after the end of the experimental period, request model information of the first electronic device together with identification information of the first electronic device to the group determining unit (203), and, upon receiving optimized experimental model information set for a plurality of experimental groups including the first electronic device from the group determining unit (201), transmit the optimized experimental model to the first electronic device (301).

[0082] According to one embodiment, the model providing unit (205) can, when confirming receipt of an update request from the plurality of electronic devices after the end of the experimental period, receive optimized experimental model information set for the plurality of groups from the group determining unit (203) and transmit the optimized experimental model to the first electronic device (301).

[0083] The experimental analysis unit (207) according to one embodiment may provide data comparing the plurality of experimental groups to the first user interface based on analysis of user usability data received from the plurality of electronic devices included in the plurality of experimental groups.

[0084] The experimental analysis unit (207) according to one embodiment may analyze user usability data received from a plurality of electronic devices included in the plurality of experimental groups, and convert the analyzed data into a numerical set, table, and / or graph and provide the result to the first user interface.

[0085] In order to verify the validity of the experiment and / or the correlation of the result data, the experiment analysis unit (207) according to one embodiment may provide a first user interface that displays statistical similarities between the plurality of experiment groups based on analysis of user characteristic data and user usability data received from the plurality of electronic devices included in the plurality of experiment groups.

[0086] The experimental analysis unit (207) according to one embodiment may analyze user characteristic data and user usability data received from a plurality of electronic devices included in the plurality of experimental groups using a T-test function and / or a statistical testing function, and provide the second user interface that displays statistical similarities between the plurality of groups based on the analysis results.

[0087] The experimental analysis unit (207) according to one embodiment may compare the average and distribution based on the first characteristic data included in the user's characteristic data when using the user's characteristic data to verify statistical similarity between multiple experimental groups.

[0088] The experimental analysis unit (207) according to one embodiment can verify the similarity of keywords with high scores by calculating a keyword set and an average value for each keyword of users of electronic devices included in each of a plurality of experimental groups based on second characteristic data included in the characteristic data of the user.

[0089] According to one embodiment, the experimental analysis unit (207) may determine an optimized experimental model for the plurality of experimental groups among the plurality of experimental models based on analysis of user usability data received from the plurality of electronic devices included in the plurality of experimental groups, and transmit the determined optimized experimental model information to the group determination unit (203).

[0090] The experimental analysis unit (207) according to one embodiment may determine an optimized experimental model for the plurality of experimental groups based on an experimental model selected through an experimenter's input while providing data comparing the plurality of experimental groups to a first user interface based on an analysis of the user's usability data received from a plurality of electronic devices included in the plurality of experimental groups, and may transmit the determined optimized experimental model information to the group determination unit (203).

[0091] In one embodiment, the first memory (220) may store commands for performing various operations in the server (210).

[0092] In one embodiment, the first memory (220) may store commands for performing a control experiment (e.g., AB test) on the server.

[0093] The first communication circuit (290) according to one embodiment may include at least one short-range communication circuit (not shown).

[0094] According to one embodiment, the first communication circuit (290) may include an NFC communication circuit, a BLE communication circuit, and / or a UWB communication circuit capable of transmitting and receiving a UWB signal with an external device using a plurality of antennas, a Wi-Fi communication circuit, and / or a Bluetooth legacy communication circuit.

[0095] In one embodiment, the server (201) provides different experimental models to multiple experimental groups and describes an operation of receiving result values ​​thereof. However, the server (201) may set multiple experimental models in one experimental group, provide multiple different experimental models to multiple electronic devices included in the one experimental group, and receive result values ​​thereof.

[0096] Among a plurality of electronic devices (310) according to an embodiment, each electronic device may transmit user characteristic data collected while a plurality of applications are running on the electronic device to the server (201). Among a plurality of electronic devices (310) according to an embodiment, each electronic device may update a first application that supports a control experiment with an experimental model received from the server (201), and transmit user usability data collected while the updated second application is being used to the server (201).

[0097] The configuration of the first electronic device (301) among the plurality of electronic devices (310) can be described in detail in FIG. 3 below, and all electronic devices included in the plurality of electronic devices (310) can have the same configuration as the first electronic device (301).

[0098] Figure 3 is a block diagram of an electronic device according to one embodiment.

[0099] Referring to the above FIG. 3, the electronic device (301) (e.g., the first electronic device (301) of FIG. 2) may include a second processor (320), a data analysis unit (340), a second memory (350), a display (360), and a second communication circuit (390).

[0100] According to one embodiment, the second processor (320) may be implemented substantially identically or similarly to the processor (120) of FIG. 1.

[0101] According to one embodiment, the second processor (320) may control the data analysis unit (340) to provide data for a control experiment to a server (e.g., server (210) of FIG. 2).

[0102] According to one embodiment, the processor (320) may perform the same function as the data analysis unit (340) or may include the data analysis unit (340).

[0103] A data analysis unit (340) according to one embodiment may collect user characteristic data collected while multiple applications are running on an electronic device periodically or at a specified time and transmit the collected data to a server (e.g., server (210) of FIG. 2).

[0104] A data analysis unit (340) according to one embodiment may store (collect) the user's characteristic data, which includes first characteristic data collected while a first application supporting a control experiment among the plurality of applications is executed (used), and second characteristic data converted into keywords from a plurality of signals (information) detected while the plurality of applications are executed on the electronic device.

[0105] According to one embodiment, the data analysis unit (340) may store (collect) at least one of the usage time, number of exposures by function, frequency of use, number of inputs by function, or amount of input data for the first application while the first application supporting the control experiment is being used as the first characteristic data.

[0106] According to one embodiment, the electronic device (301) may include at least one first application that supports a control experiment among a plurality of applications.

[0107] According to one embodiment, the data analysis unit (340) may include a plurality of signals (information) detected while the plurality of applications are running, such as screen information of the electronic device (301), input information (e.g., touch, keyboard and / or mouse), log information recording the usage history of the application and / or function, sound information, location information and / or motion information.

[0108] According to one embodiment, the data analysis unit (340) detects screen information, input information (e.g., touch, keyboard and / or mouse, etc.), log information recording the usage history of applications and / or functions, sound information, location information and / or motion information of the electronic device (301), and may convert the detected information into keywords using a keyword conversion function such as an OCR function, an image recognition function and / or a voice recognition function.

[0109] According to one embodiment, the data analysis unit (340) may convert a plurality of signals (information) detected while the electronic device is being used into a first keyword using the keyword conversion function, and may calculate at least one second keyword related to the first keyword and a score for the at least one second keyword using the keyword distance dictionary stored in the second memory (330).

[0110] According to one embodiment, the keyword distance dictionary may be a graph model having a data structure defining the distance (score) between the keyword and a keyword (word) that can define the user's specificity. In the distance keyword distance dictionary according to one embodiment, the distance between keywords may be expressed numerically.

[0111] The data analysis unit (340) according to one embodiment may convert information detected while the electronic device is being used into a first keyword using a keyword conversion function, and identify at least one second keyword that satisfies a keyword processing condition among a plurality of keywords included in a keyword distance dictionary. The data analysis unit (340) according to one embodiment may calculate a score of the second keyword using a distance (number) between the first keyword and the at least one second keyword in the keyword distance dictionary, and collect (generate) second characteristic data including the at least one second keyword and the score of the second keyword.

[0112] In order to prevent complete exposure of personal information, the data analysis unit (340) according to one embodiment may collect (generate) second characteristic data including at least one second keyword calculated based on a keyword distance dictionary and a score of the at least one second keyword, without including the first keyword in the second characteristic data, which is converted using a keyword conversion function from information detected while the electronic device is in use.

[0113] For example, when the data analysis unit (340) detects a first keyword, "meeting," using a voice recognition function based on sound information detected while the electronic device is in use, it can identify a plurality of second keywords, such as "work" having a distance value of "1" from the first keyword "meeting" that satisfies a keyword processing condition (e.g., condition = distance < 3, score = 1 / distance) in a keyword distance dictionary, and "boss" and "customer" having a distance value of "2" from the first keyword "meeting." The data analysis unit (340) can calculate scores of the plurality of second keywords using the distances (numbers) between the first keyword ("meeting") and the plurality of second keywords ("work," "boss," and "customer"), and collect (generate) second characteristic data including a plurality of second keywords, such as "work = 1, boss = 0.5, customer = 0.5," and scores of the plurality of second keywords, as user characteristic data.

[0114] According to one embodiment, the second processor (320) or data analysis unit (340) may request an update inquiry for the first application supporting the control experiment from a server (e.g., server (210) of FIG. 2), and receive a model capable of updating the first application in response to the request.

[0115] According to one embodiment, the second processor (320) or the data analysis unit (340) may periodically or at a designated time request an update inquiry for the first application along with identification information of the first application that supports a control experiment to the server.

[0116] According to one embodiment, the second processor (320) or the data analysis unit (340), if the model received from the server is confirmed to be an experimental model, may update the first application with the experimental model, collect and store the user's usage data while the updated first application is executed (used), and transmit the user's usage data to the server.

[0117] According to one embodiment, the second processor (320) may, upon confirming the selection of the first application updated with the experimental model, display content based on the update information of the first application through the display (260), and collect and store usability information of a user using the first application displaying the content.

[0118] According to one embodiment, the second processor (320) or the data analysis unit (340) may collect and store the usage period, usage frequency, and / or amount of input data of the first application updated as an experimental model as the user's usability data.

[0119] According to one embodiment, the second processor (320) or the data analysis unit (340) may update the first application with the model if the model received from the server is confirmed to be a model set in a group including the electronic device or a basic model.

[0120] The second memory (350) according to one embodiment may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0121] In one embodiment, the second memory (350) may store a plurality of applications including at least one first application that supports a control experiment (e.g., AB test).

[0122] According to one embodiment, the second memory (350) may store user characteristic data collected while the electronic device is in use and user usability data collected while the application updated with the experimental model is in use.

[0123] According to one embodiment, the second memory (350) may store user characteristic data collected while the electronic device is in use and user usability data collected while the application updated with the experimental model is in use.

[0124] According to one embodiment, the display (360) may be implemented substantially identically or similarly to the display (160) of FIG. 1.

[0125] According to one embodiment, the display (360) may display content based on update information of the first application.

[0126] According to one embodiment, the second communication circuit (390) may be implemented substantially identically or similarly to the communication module (190) of FIG. 1.

[0127] According to one embodiment, the second communication circuit (390) may include at least one of a wireless LAN circuit (not shown) and a short-range communication circuit (not shown).

[0128] The second communication circuit (390) according to one embodiment may include an NFC communication circuit, a BLE communication circuit, and / or a UWB communication circuit capable of transmitting and receiving a UWB signal with an external device using a plurality of antennas, a Wi-Fi communication circuit, and / or a Bluetooth legacy communication circuit.

[0129] FIG. 4 is a diagram for explaining an operation of determining multiple experimental groups based on user characteristic data in a server according to an embodiment of the present invention.

[0130] Referring to the above FIG. 4, the user characteristic data including "10 hours" as the first characteristic data, which represents the usage time of the first application supporting the control experiment collected by the data analysis unit of the first electronic device (301) among the plurality of electronic devices (310) for a specified period of time, is transmitted to the group management unit (201) of the server (e.g., the server (210) of FIG. 2), the user characteristic data including "1 hour" as the first characteristic data, which represents the usage time of the first application supporting the control experiment collected by the data analysis unit of the second electronic device (303), is transmitted to the group management unit (201) of the server (e.g., the server (210) of FIG. 2), and the user characteristic data including "0.5 hours" as the first characteristic data, which represents the usage time of the first application supporting the control experiment collected by the data analysis unit of the third electronic device (303) is transmitted to the group management unit (201) of the server (210) (e.g., the server (210) of FIG. 2)) It can be transmitted to the management department (201).

[0131] The group management unit (201) of the server (210) may include a receiving unit (202), a classification unit (205), and a storage unit (207). Data stored in the storage unit (207) may be stored in the same manner in the first memory of the server (e.g., the first memory (250) of FIG. 2), and the storage unit (207) may be at least partially included in the first memory.

[0132] The receiving unit (202) can receive first characteristic data (first electronic device = usage time 10, second electronic device = usage time 1, third electronic device = usage time 0.5) of the user's characteristic data from the first electronic device (301), the second electronic device (303), and the third electronic device (305). The receiving unit (202) can store the user's characteristic data (first electronic device = usage time 10, second electronic device = usage time 1, third electronic device = usage time 0.5) in the first area (207a) (datalake) of the storage unit (207) corresponding to the identification information of the electronic devices, and transmit the first characteristic data of the user's characteristic data to the classification unit (205).

[0133] When the above classification unit (205) receives the first characteristic data (first electronic device = usage time 10, second electronic device = usage time 1, third electronic device = usage time 0.5) of the user's characteristic data from the constant receiving unit (202), the classification unit (205) can classify the first electronic device (301) into a heavy group by referring to the group conditions stored in the second area (207b) of the storage unit (207), classify the second electronic device (303) and the third electronic device (305) into a light group, and then store information of the classified group in the third area (207c) of the storage unit (207).

[0134] FIGS. 5A, 5B, and 5C are diagrams illustrating an operation of determining multiple experimental groups based on user characteristic data by a server and an electronic device according to one embodiment.

[0135] Referring to the above FIG. 5A, the electronic device (301) (e.g., the electronic device (301) of FIG. 3) can receive, through the framework (380), a plurality of signals (information) detected while a plurality of applications of the electronic device (301) are being executed, for example, screen information, input information (e.g., touch, keyboard and / or mouse, etc.), log information recording the usage history of applications and / or functions, sound information, location information and / or motion information.

[0136] In operation 501, the electronic device (301) can convert a plurality of signals (information) detected while a plurality of applications are running in the electronic device (301) into a first keyword using a keyword conversion function such as an OCR function, an image recognition function, and / or a voice recognition function.

[0137] If the electronic device (301) confirms the converted first keywords as “meeting” and “office” in operation 503, the electronic device (301) can check the keyword processing conditions for processing the keywords in operation 505, and calculate a second keyword that satisfies the keyword processing conditions from a keyword distance dictionary (530) stored in a memory (e.g., memory (330) of FIG. 3) of the electronic device (301). The electronic device (301) can check “meeting” (a1) among “meeting” and “office” in the keyword distance dictionary, and check a plurality of second keywords such as “work” (b1), “boss” (b2), and “customer” (b3) whose distance values ​​are “3” or less based on “meeting” (a1), and check a score by calculating the distance value of each of the plurality of second keywords as “1 / distance”.

[0138] The electronic device (301), in operation 507, can confirm that the first keyword “meeting” has an infinite value, the score value of the plurality of second keywords “work” is “1”, the score value of “boss” is “0.5”, and the score value of “customer” is “0.5”, and transmit the user’s characteristic data including the plurality of second keywords and the scores of the plurality of second keywords (“work = 1”, “boss = 0.5”, and “customer = 0.5”) as second characteristic data to a server (e.g., server (210) of FIG. 2).

[0139] Referring to the above FIG. 5b, the group management unit (201) (e.g., the group management unit (201) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) may include a receiving unit (202), a classification unit (205), and a storage unit (207).

[0140] When the receiving unit (202) receives second characteristic data (“work = 1”, “superior = 0.5” and “customer = 0.5”) among the user’s characteristic data from the electronic device (301), the receiving unit (202) stores the second characteristic data in the first area (207a) (datalake) of the storage unit (207) corresponding to the identification information of the electronic device, and transmits the second characteristic data of the user’s characteristic data to the classification unit (205).

[0141] The above classification unit (205), when receiving the second characteristic data of the user's characteristic data ("work = 1", "superior = 0.5" and "customer = 0.5") from the constant receiving unit (202), classifies the electronic device (301) into one of a plurality of groups by referring to the group conditions stored in the second area (207b) of the storage unit (207), and stores information of the classified group of the electronic device (301) in the third area (207c) of the storage unit (207).

[0142] The group determination unit (203) (e.g., the group determination unit (203) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) requests group information corresponding to the experimental conditions to the group management unit (201) based on the experimental conditions input by the experimenter, and can receive information on group A and information on group B from the group management unit (201). The group determination unit (203) can determine an experimental group A1 including n electronic devices among a plurality of electronic devices included in one group A and an experimental group A2 including x electronic devices. The group determination unit (203) can determine an experimental group B1 including y electronic devices among a plurality of electronic devices included in group B and an experimental group B2 including z electronic devices among a plurality of electronic devices included in group B.

[0143] The group determination unit (203) can determine experimental groups A1, A2, B1, and B2 for executing experiments entered by the experimenter during a specified period. Information related to the plurality of experimental groups, i.e., experimental groups A1, A2, B1, and B2, can be stored in a table (551) in the storage unit (550) of the group determination unit (203).

[0144] Referring to the above FIG. 5c, the group determination unit (203) stores information related to the plurality of experimental groups, i.e., experimental group A1, experimental group A2, experimental group B1, and experimental group B2, in a table (551) in the storage unit (550) of the group determination unit (203) (e.g., a list of users selected for each group), and can store model N information set in experimental group A1, model X information set in experimental group A2, model Y information set in experimental group B1, and model Z information set in experimental group B2 in a table (553).

[0145] When the model providing unit (205) of the server (210) receives an update request of a first application supporting a control experiment from an electronic device (301) (e.g., the processor (320) of the electronic device (301) of FIG. 3 (e.g., the second processor (320) of FIG. 3)) (511), the model providing unit (205) of the server (210) may request the identification information of the electronic device (301) and model information set for a group including the electronic device (301) to the group determining unit (203). When the model providing unit (205) receives model information of an experimental group including the electronic device (301) from the group determining unit (203) (515), the model providing unit (205) may transmit model information to the electronic device (301) (517). When the model providing unit (205) receives a download request from the electronic device (301) (519), the model of the experimental group including the electronic device (301) may be transmitted to the electronic device (301). There is (521).

[0146] The processor (320) of the electronic device (301) receives an experimental model from the server (210) and can update the first application (395) with the received experimental model (523).

[0147] FIGS. 6A and 6B are diagrams illustrating an operation of multiple experimental groups executing an application in an electronic device according to one embodiment.

[0148] In order to conduct an experiment to determine whether an "extroverted user" "prefers sharing content in an application" and "prefers expressions such as comments or reactions," the group determination unit (203) (e.g., the group determination unit (203) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) may receive information about a plurality of groups related to the "extroverted user" among a plurality of groups managed by the group management unit (e.g., the group management unit (201) of FIG. 2) of the server (210).

[0149] Referring to the above Fig. 6a, the group determination unit (203) can determine experimental group A, experimental group B, and experimental group C among the multiple groups received from the group management unit. The group determination unit (203) can set model 1 that can "show sharing to other users first" for experimental group A, model 2 that can "show comments and recommendation functions to users first" for experimental group B, and model 3 that can "show randomly" for experimental group C.

[0150] The model providing unit (205) (e.g., the model providing unit (205) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) can transmit model 1 to the first electronic device (301) included in the experimental group A, transmit model 2 to the second electronic device (302) included in the experimental group B, and transmit model 3 to the first electronic device (301) included in the experimental group C, based on the information on the plurality of experimental groups and the information on the plurality of models received from the group determining unit (203).

[0151] The first electronic device (301) updates a first application capable of displaying content to model 1, and upon confirming execution of the updated first application, displays content (611) on the display of the first electronic device (301), and displays a function UI (613a) sequentially displaying a first function for sharing the content (611) with other users and a second function for comments and recommendations.

[0152] The second electronic device (303) updates the first application capable of displaying content to model 2, and upon confirming execution of the updated first application, displays content (611) on the display of the second electronic device (303) and displays a function UI (613b) sequentially displaying a second function for comments and recommendations on the content and a first function for sharing with other users.

[0153] The third electronic device (303) updates the first application capable of displaying content to model 3, and upon confirming execution of the updated first application, displays content (611) on the display of the third electronic device (305), and can display a function UI (613c) that randomly displays a first function for sharing with other users and a second function for comments and recommendations for the content (611).

[0154] Referring to the above Fig. 6b, the group determination unit (203) can determine experimental group A, experimental group B, and experimental group C among the multiple groups received from the group management unit. The group determination unit (203) can set model 1, which can "display content in the first order," for experimental group A, model 2, which can "display content in the second order," for experimental group B, and model 3, which can "display content randomly," for experimental group C.

[0155] The model providing unit (205) (e.g., the model providing unit (205) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) can transmit model 1 to the first electronic device (301) included in the experimental group A, transmit model 2 to the second electronic device (302) included in the experimental group B, and transmit model 3 to the first electronic device (301) included in the experimental group C, based on the information on the plurality of experimental groups and the information on the plurality of models received from the group determining unit (203).

[0156] The first electronic device (301) updates a first application capable of displaying content to model 1, and when the execution of the updated first application is confirmed, the first electronic device (301) may display content (611a) in a first order on the display of the first electronic device (301), and may display a function UI (613a) that sequentially displays a first function for sharing with other users and a second function for comments and recommendations for the content (611a) displayed in the first order. The first electronic device (301) may collect (store) usage time, usage frequency, number of inputs, and / or amount of input data for the first application as user usage data, and transmit the collected user usage data to the server.

[0157] The second electronic device (303) updates the first application capable of displaying content to model 2, and upon confirming execution of the updated first application, displays content (611b) in a second order on the display of the second electronic device (303), and displays a function UI (613a) that sequentially displays a first function for sharing with other users and a second function for comments and recommendations for the content (611b) displayed in the second order. The second electronic device (303) may collect (store) the usage time, usage frequency, number of inputs, and / or amount of input data for the first application as the user's usage data, and transmit the collected user's usage data to the server.

[0158] The third electronic device (303) updates the first application capable of displaying content to model 3, and when the execution of the updated first application is confirmed, the third electronic device (305) can randomly display content (611c) on the display, and can display a function UI (613a) that sequentially displays a first function for sharing with other users and a second function for comments and recommendations for the randomly displayed content (611c). The third electronic device (303) can collect (store) the usage time, usage frequency, number of inputs, and / or amount of input data for the first application as the user's usage data, and transmit the collected user's usage data to the server.

[0159] FIGS. 7A and 7B are diagrams illustrating experimental results through multiple experimental groups in a server according to one embodiment.

[0160] As shown in the above FIG. 7a, the experimental analysis unit (e.g., the experimental analysis unit (207) of FIG. 2) of the server (e.g., the server (210) of FIG. 2) may, upon confirming a request for experimental result data based on an experimenter's input, analyze the user's usability data received from a plurality of electronic devices included in the plurality of experimental groups, and convert and process the analyzed data into a numerical set, a table, and / or a graph and provide the result to the first user interface (711). Upon confirming the selection (711a) of the application usage time in the first user interface (711), the experimental analysis unit may compare the sum of the application usage times on the plurality of electronic devices included in the experimental group A during a specified period from October 14, 2023 to October 16, 2023, and display the result as a graph (711b) and a table (711c).

[0161] As shown in the above FIG. 7b, the experiment analysis unit may provide a second user interface (713) that displays statistical similarities between the plurality of experiment groups based on analysis of user characteristic data and user usability data received from a plurality of electronic devices included in the plurality of experiment groups in order to verify the validity of the experiment and / or the correlation of the result data. The experiment analysis unit may display the statistical similarities between the experiment group A and the experiment group B as a graph (713a) on the second user interface (713).

[0162] An electronic device (the first electronic device (101) of FIG. 1 and / or the electronic device (301) of FIG. 3) according to an embodiment may include a communication circuit (the communication circuit (190) of FIG. 1 and / or the second communication circuit (390) of FIG. 3), at least one processor (the processor (120) of FIG. 1 and / or the second processor (320) of FIG. 3)) and a memory (the memory (130) of FIG. 1 and / or the second memory (330) of FIG. 3) storing instructions. The instructions according to an embodiment may be configured to cause the electronic device, when executed by the at least one processor, to transmit user characteristic data collected while a plurality of applications are executed in the electronic device to a server. The instructions according to an embodiment may be configured to cause the electronic device, when receiving an experiment model determined based on at least a portion of the user characteristic data from the server, to update a first application among the plurality of applications using the received experiment model. The instructions according to one embodiment, when executed by the at least one processor, may be configured to cause the electronic device to transmit the user's usage data collected while the updated first application is running to the server.

[0163] The instructions according to one embodiment may be configured to cause the electronic device to transmit, to the server, user characteristic data including first characteristic data collected while the first application is executed and second characteristic data obtained by converting a plurality of signals detected while the plurality of applications are executed into keywords, when executed by the at least one processor.

[0164] A server (server 108 of FIG. 1 and / or server 210 of FIG. 2) according to an embodiment may include a communication circuit (first communication circuit 290 of FIG. 2), at least one processor (first processor 220 of FIG. 2), and a memory (first memory 250 of FIG. 2) storing instructions. The instructions according to an embodiment may be configured to cause the electronic devices to determine a plurality of experimental groups among the plurality of electronic devices based on user characteristic data received from the plurality of electronic devices when executed by the at least one processor. The instructions according to an embodiment may be configured to cause the electronic devices to determine a plurality of experimental models associated with applications to be set for each of the plurality of experimental groups when executed by the at least one processor. In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to transmit a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups. In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to transmit a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups. In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to determine an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user's usability data received from the plurality of experimental groups.

[0165] In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to classify the plurality of electronic devices into a plurality of groups based on the user characteristic data received from the plurality of electronic devices. In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to determine the plurality of experimental groups from among the plurality of groups based on a specified condition.

[0166] The instructions according to one embodiment may be configured to cause the electronic devices, when executed by the at least one processor, to determine the plurality of experimental groups, including a first experimental group identified from a first group among the plurality of groups and a second experimental group identified from a second group.

[0167] The instructions according to one embodiment, when executed by the at least one processor, may be configured to cause the electronic devices to determine the plurality of experimental groups, including a first experimental group and a second experimental group identified from a first group among the plurality of groups.

[0168] The plurality of experimental models according to one embodiment may differ in at least one of properties, setting variables, functions, UI elements, statistical models, or algorithms.

[0169] The instructions according to one embodiment may be configured to, when executed by the at least one processor, cause the electronic devices to confirm receipt of an application update request from a first electronic device among the plurality of electronic devices. The instructions according to one embodiment may be configured to, when executed by the at least one processor, cause the electronic devices to, when it is confirmed based on identification information of the first electronic device that the first electronic device is included in a first experimental group among the plurality of experimental groups, transmit a first experimental model set in the first experimental group among the plurality of experimental models to the first electronic device.

[0170] In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to transmit a model set for a third group to the first electronic device if the first electronic device is determined to be included in a third group among the plurality of experimental groups based on identification information of the first electronic device. In one embodiment, the commands, when executed by the at least one processor, may be configured to cause the electronic devices to transmit a default model to the first electronic device if a model for the third group including the first electronic device is not set.

[0171] The instructions according to one embodiment may be configured to cause the electronic devices, when executed by the at least one processor, to provide a first user interface for displaying data comparing the plurality of experimental groups based on an analysis of the user's usability data and a second user interface for displaying statistical similarities between the plurality of experimental groups based on an analysis of the user's characteristic data and the user's usability data.

[0172] FIGS. 8A, 8B, and 8C are flowcharts illustrating operations for performing a control experiment by a server and an electronic device according to an embodiment. The operations for performing the control experiment by the server and the electronic device may include operations 801 to 841. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0173] In operation 801, a first electronic device among a plurality of electronic devices (e.g., the electronic device (101) of FIG. 1, the first electronic device (301) of FIG. 2, and / or the electronic device (301) of FIG. 3) may collect user characteristic data.

[0174] According to one embodiment, the first electronic device (301) can collect (store) user characteristic data collected while a plurality of applications are running on the first electronic device periodically or at a specified time.

[0175] According to an embodiment, the first electronic device (301) may store (collect) the user's characteristic data, which includes first characteristic data collected while a first application supporting a control experiment among a plurality of applications is executed (used), and second characteristic data obtained by converting a plurality of signals (information) detected while a plurality of applications are executed on the first electronic device into keywords.

[0176] According to one embodiment, the first electronic device (301) may store (collect) at least one of the usage time, number of exposures per function, frequency of use, number of inputs per function, or amount of input data for the first application as the first characteristic data while the first application supporting the control experiment is being executed (used).

[0177] According to one embodiment, the first electronic device (301) may include at least one first application that supports a control experiment among a plurality of applications.

[0178] According to one embodiment, the plurality of signals (information) detected while a plurality of applications are running in the first electronic device may include screen information of the first electronic device (301), input information (e.g., touch, keyboard and / or mouse, etc.), log information recording the usage history of applications and / or functions, sound information, location information and / or motion information.

[0179] According to one embodiment, the first electronic device (301) may detect screen information, input information (e.g., touch, keyboard and / or mouse, etc.), log information recording the usage history of applications and / or functions, sound information, location information and / or motion information of the first electronic device (301), and convert the detected information into keywords using a keyword conversion function such as an OCR function, an image recognition function and / or a voice recognition function.

[0180] According to one embodiment, the first electronic device (301) can convert information detected while the first electronic device is being used into a first keyword using a keyword conversion function, and can identify at least one second keyword that satisfies a keyword processing condition among a plurality of keywords included in a keyword distance dictionary.

[0181] According to one embodiment, the first electronic device (301) may calculate a score of the second keyword using a distance (number) between the first keyword and the at least one second keyword in the keyword distance dictionary, and collect (generate) second characteristic data including the at least one second keyword and the score of the second keyword.

[0182] In order to prevent complete exposure of personal information, the first electronic device (301) according to one embodiment may collect (generate) second characteristic data including at least one second keyword calculated based on a keyword distance dictionary and a score of the at least one second keyword, without including the first keyword converted using a keyword conversion function from information detected while the first electronic device is in use in the second characteristic data.

[0183] In operation 803, the first electronic device (301) can transmit the user's characteristic data to the group management unit (201) (e.g., the group management unit (201) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2).

[0184] In operation 805, the group management unit (201) can classify and manage multiple electronic devices (310) into multiple groups based on the user's usability data.

[0185] According to one embodiment, the group management unit (201) receives user characteristic data from a plurality of electronic devices (310) including the first electronic device (301), stores the user characteristic data, and manages the plurality of electronic devices (310) by classifying them into a plurality of groups based on the user characteristic data.

[0186] In operation 807, the group determination unit (203) (e.g., the group determination unit (203) of FIG. 2) of the server (210) (e.g., the server (210) of FIG. 2) may request multiple group information from the group management unit (201), and in operation 809, may receive multiple group information from the group management unit (201).

[0187] In operation 811, the group determination unit (203) can determine multiple experimental groups among multiple groups.

[0188] A group determination unit (203) according to one embodiment can determine a plurality of experimental groups for a control experiment among a plurality of groups.

[0189] According to one embodiment, the group determination unit (203) can randomly determine multiple experimental groups for a control experiment among multiple groups.

[0190] According to one embodiment, the group determination unit (203) may determine a first group and a second group having user characteristic data that meet the experimental conditions among the plurality of groups based on the experimental conditions input by the experimenter, determine a first experimental group from the first group, and determine a second experimental group from the second group.

[0191] The group determination unit (203) according to one embodiment may determine a first group and a second group having user characteristic data that meet the experimental conditions among the plurality of groups based on the experimental conditions input by the experimenter, and may determine a first experimental group and a second experimental group from the first group, and / or determine a first experimental group and a second experimental group from the second group.

[0192] According to an embodiment, the group determination unit (203) may determine a plurality of experimental models related to a first application to be set in each of the plurality of experimental groups. The group determination unit (203) may determine a first experimental model among the plurality of experimental models in a first experimental group including a plurality of first electronic devices among the plurality of electronic devices as an experimental model related to the first application. The group determination unit (203) may determine a second experimental model among the plurality of experimental models in a second experimental group including a plurality of second electronic devices among the plurality of electronic devices as an experimental model related to the first application.

[0193] The group decision unit (203) according to one embodiment may differently change at least one of the properties, setting variables, functions, UI elements, statistical models or algorithms included in the plurality of experimental models in order to update the first application under different conditions.

[0194] In operation 813, the first electronic device (301) may request an update inquiry for the first application to the model providing unit (205) of the server (210) (e.g., the model providing unit (205) of FIG. 2).

[0195] According to one embodiment, the first electronic device (301) may periodically or at a specified time request an update inquiry for the first application along with identification information of the first application supporting the control experiment to the server.

[0196] In operation 815, the model providing unit (205) can check the model set in the experimental group including the first electronic device among the plurality of experimental groups.

[0197] A model providing unit (205) according to one embodiment can transmit experimental models set to a plurality of electronic devices included in a plurality of experimental groups.

[0198] According to one embodiment, the model providing unit (205) may request model information set for a group including the first electronic device together with identification information of the first electronic device from the group determining unit (203), and receive, from the group determining unit, first experimental model information set for a first experimental group including the electronic device (301) among the plurality of experimental groups.

[0199] In operation 817, the model providing unit (205) can transmit an experimental model for updating the first application.

[0200] In operation 819, the first electronic device (301) can update the first application and collect the user's usage data while the updated first application is being executed (used).

[0201] According to one embodiment, the first electronic device (301), if the model received from the model provision unit (207) of the server (210) is confirmed as an experimental model, updates the first application with the experimental model, and collects (stores) the user's usability data while the updated first application is being used.

[0202] According to one embodiment, the first electronic device (301) can, upon confirming selection of a first application updated with an experimental model, display content based on update information of the first application through a display of the first electronic device, and collect (store) usability information of a user using the first application displaying the content.

[0203] According to one embodiment, the first electronic device (301) can collect (store) the usage period, usage frequency and / or amount of input data of the first application updated as an experimental model as the user's usability data.

[0204] In operation 821, the first electronic device (301) can transmit the user's usability data to the group management unit (201) of the server (210).

[0205] In operation 823, the group management unit (201) can store the user's usability data.

[0206] In one embodiment, the group management unit (201) may receive user usage data collected while a first application updated with an experimental model is executed (used) from a plurality of electronic devices (310) included in a plurality of experimental groups during a control experimental period, and store the user usage data for each of the plurality of experimental groups.

[0207] In operation 825, the experimental analysis unit (207) of the server (210) (e.g., the experimental analysis unit (207) of FIG. 2) can confirm a request for experimental results.

[0208] The experimental analysis unit (207) according to one embodiment can confirm a request for experimental results based on the experimenter's input.

[0209] In operation 827, the experimental analysis unit (207) may request information on a plurality of experimental groups and information on experimental models set for each of the plurality of experimental groups from the group determination unit (203), and in operation 829, may receive information on a plurality of experimental groups and information on experimental models set for each of the plurality of experimental groups from the group determination unit (203).

[0210] In operation 831, the experimental analysis unit (207) may request the group management unit (201) to provide the user's usability data stored for each of a plurality of experimental groups, and in operation 833, may receive the user's usability data stored for each of a plurality of experimental groups from the group management unit (201).

[0211] In operation 835, the experimental analysis unit (207) may provide a user interface that displays the experimental results.

[0212] The experimental analysis unit (207) according to one embodiment may provide data comparing the plurality of groups to the first user interface based on analysis of user usability data received from the plurality of electronic devices included in the plurality of experimental groups.

[0213] The experimental analysis unit (207) according to one embodiment may analyze user usability data received from a plurality of electronic devices included in the plurality of experimental groups, and convert the analyzed data into a numerical set, table, and / or graph and provide the result to the first user interface.

[0214] In order to verify the validity of the experiment and / or the correlation of the result data, the experiment analysis unit (207) according to one embodiment may provide a second user interface that displays statistical similarities between the plurality of experiment groups based on analysis of user characteristic data and user usability data received from the plurality of electronic devices included in the plurality of experiment groups.

[0215] The experimental analysis unit (207) according to one embodiment may analyze user characteristic data and user usability data received from a plurality of electronic devices included in the plurality of experimental groups using a T-test function and / or a statistical testing function, and provide the second user interface that displays statistical similarities between the plurality of groups based on the analysis results.

[0216] In operation 837, the experimental analysis unit (207) can determine an experimental model optimized for the plurality of experimental groups among the plurality of experimental models.

[0217] According to one embodiment, the experimental analysis unit (207) may determine an optimized experimental model for the plurality of experimental groups among the plurality of experimental models based on analysis of user usability data received from the plurality of electronic devices included in the plurality of experimental groups.

[0218] The experimental analysis unit (207) according to one embodiment may determine an optimized experimental model for the plurality of experimental groups based on an experimental model selected through an input of an experimenter while providing data comparing the plurality of experimental groups to a first user interface based on an analysis of user usability data received from a plurality of electronic devices included in the plurality of experimental groups.

[0219] In operation 839, the experimental analysis unit (207) can transmit information of the optimized experimental model to the group decision unit (203).

[0220] In operation 841, the group decision unit (203) can set multiple experimental groups to the optimized experimental model.

[0221] According to one embodiment, the group determination unit (203) may, when receiving information on an optimized experimental model for the plurality of experimental groups from the experimental analysis unit (207) after the experimental period ends, set the optimized experimental model as a model for the plurality of experimental groups.

[0222] According to one embodiment, the group decision unit (203) may request information on multiple groups from the group management unit (201) and set the optimized experimental model as a model of the multiple groups.

[0223] Figure 9 is a flowchart illustrating the operation of providing a model from a server according to one embodiment. The operation of providing a model from the server may include operations 901 to 913. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, at least two operations may be performed in parallel, or additional operations may be added.

[0224] In operation 901, a model providing unit (e.g., a model providing unit (205) of FIG. 2) of a server (e.g., a server (210) of FIG. 2) can confirm receipt of an update request from a first electronic device (e.g., a first electronic device (301) of FIG. 2 and / or an electronic device (301) of FIG. 3) among a plurality of electronic devices (e.g., a plurality of electronic devices (310) of FIG. 2).

[0225] In operation 903, the model providing unit of the server can check whether the first electronic device is included in a plurality of experimental groups.

[0226] In operation 903, if the model providing unit of the server confirms that the first electronic device is included in the plurality of experimental groups based on the identification information of the first electronic device, in operation 905, the experimental model set in the experimental group included in the first electronic device can be transmitted to the first electronic device.

[0227] According to one embodiment, the model providing unit of the server may request model information set for a group including the first electronic device together with identification information of the first electronic device to the group determining unit of the server (e.g., the group determining unit (203) of FIG. 2).

[0228] According to one embodiment, the model providing unit of the server may, when receiving first experiment group information including the first electronic device among the plurality of experiment groups and first experiment model information set in the first experiment group from the group determining unit of the server, transmit the first experiment model to the first electronic device.

[0229] In operation 903, if the model providing unit of the server verifies that the first electronic device is not included in the plurality of experimental groups based on the identification information of the first electronic device, then in operation 907, it can verify that the first electronic device is included in the plurality of groups that are not included in the plurality of experimental groups.

[0230] According to one embodiment, the model providing unit of the server may, when receiving third group information including the first electronic device among the plurality of groups from the group determining unit of the server, confirm that the first electronic device is included in the third group among the plurality of groups.

[0231] In operation 909, the model providing unit of the server can check whether a model is set in a group included in the first electronic device among a plurality of groups.

[0232] According to one embodiment, the model providing unit of the server may, when receiving, from the group determining unit of the server, information on a third group including the first electronic device among the plurality of groups and information on a model set for the third group, confirm that a model for the third group is set.

[0233] In the above operation 909, the model providing unit of the server can, if it confirms the setting of the model for the group included in the first electronic device, transmit the model set for the group included in the first electronic device to the first electronic device in operation 911.

[0234] According to one embodiment, the model providing unit of the server can transmit a model, in which a model is set in the third group, to the first electronic device.

[0235] In the above operation 909, if the model providing unit of the server confirms that a model for a group included in the first electronic device is not set, in operation 913, the basic model can be transmitted to the first electronic device.

[0236] In one embodiment, the model providing unit of the server may, when receiving, from the group determining unit of the server, information indicating that there is no model set for the third group and information regarding a third group including the first electronic device among the plurality of groups, transmit a basic model to the first electronic device.

[0237] A method of operating an electronic device according to one embodiment may include transmitting user characteristic data collected while a plurality of applications are running on the electronic device to a server. The method according to one embodiment may include receiving an experimental model determined based on at least a portion of the user characteristic data from the server, and updating a first application among the plurality of applications using the received experimental model. The method according to one embodiment may include transmitting user usage data collected while the updated first application is running to the server.

[0238] The method according to one embodiment may further include transmitting, to the server, the user's characteristic data, which includes first characteristic data collected while the first application is executed and second characteristic data converted into keywords from a plurality of signals detected while the plurality of applications are executed.

[0239] According to one embodiment, a method of operating a server may include an operation of determining a plurality of experimental groups among a plurality of electronic devices based on user characteristic data received from the plurality of electronic devices. According to one embodiment, the method may include an operation of determining a plurality of experimental models related to an application to be set for each of the plurality of experimental groups. According to one embodiment, the method may include an operation of transmitting a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups. According to one embodiment, the method may include an operation of transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups. According to one embodiment, the method may include an operation of determining an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user's usability data received from the plurality of experimental groups.

[0240] The method according to one embodiment may include an operation of classifying the plurality of electronic devices into a plurality of groups based on the user characteristic data received from the plurality of electronic devices. The method according to one embodiment may further include an operation of determining the plurality of experimental groups among the plurality of groups based on a specified condition.

[0241] The method according to one embodiment may further include an operation of determining the plurality of experimental groups, including a first experimental group identified in a first group among the plurality of groups and a second experimental group identified in a second group.

[0242] The method according to one embodiment may further include an operation of determining the plurality of experimental groups including a first experimental group and a second experimental group identified in a first group among the plurality of groups.

[0243] The plurality of experimental models according to one embodiment may differ in at least one of properties, setting variables, functions, UI elements, statistical models, or algorithms.

[0244] The method according to one embodiment may include an operation of confirming receipt of an application update request from a first electronic device among the plurality of electronic devices. The method according to one embodiment may further include an operation of transmitting a first experimental model set in the first experimental group among the plurality of experimental models to the first electronic device when it is confirmed based on identification information of the first electronic device that the first electronic device is included in a first experimental group among the plurality of experimental groups.

[0245] In one embodiment, the method may include an operation of transmitting a model set for the third group to the first electronic device when it is confirmed based on identification information of the first electronic device that the first electronic device is included in a third group that is not included in the plurality of experimental groups among the plurality of groups. In one embodiment, the method may further include an operation of transmitting a default model to the first electronic device when a model for the third group including the first electronic device is not set.

[0246] The method according to one embodiment may further include providing a first user interface for displaying data comparing the plurality of experimental groups based on analysis of the user's usability data, and a second user interface for displaying statistical similarities between the plurality of experimental groups based on analysis of the user's characteristic data and the user's usability data.

[0247] Electronic devices according to embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0248] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0249] The term "module" used in one embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0250] An embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101) or an electronic device (301)). For example, a processor (e.g., a processor (520)) of the machine (e.g., an electronic device (301)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0251] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be available through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0252] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101 in FIG. 1; 301 in FIG. 3), Communication circuit (190 in Fig. 1: 390 in Fig. 3); At least one processor (120 in FIG. 1; 320 in FIG. 3); and It includes a memory (130 in Fig. 1; 330 in Fig. 3) that stores commands, The above instructions, when executed by the at least one processor, cause the electronic device to: Transmitting user characteristic data collected while multiple applications are running on the electronic device to the server, Upon receiving an experimental model determined based on at least a portion of the user's characteristic data from the server, updating a first application among the plurality of applications using the received experimental model, An electronic device configured to transmit user usage data collected while the above updated first application is running to the server.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device configured to transmit the user's characteristic data, including first characteristic data collected while the first application is running and second characteristic data converted into keywords from a plurality of signals detected while the plurality of applications are running, to the server.

3. In the server (108 in Fig. 1; 210 in Fig. 2), Communication circuit (290 in Fig. 2); At least one processor (220 in FIG. 2); and It includes a memory (250 in Fig. 2) that stores commands, The above instructions, when executed by the at least one processor, cause the electronic devices to: Determine multiple experimental groups among the multiple electronic devices based on user characteristic data received from the multiple electronic devices, Determine multiple experimental models related to the application to be set for each of the multiple experimental groups above, A plurality of first electronic devices included in a first experimental group among the plurality of experimental groups transmit a first experimental model among the plurality of experimental models, Transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups, A server configured to determine an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user usability data received from the plurality of experimental groups.

4. In paragraph 3, The above instructions, when executed by the at least one processor, cause the server to: Classifying the plurality of electronic devices into a plurality of groups based on the user characteristic data received from the plurality of electronic devices, A server set to determine the plurality of experimental groups among the plurality of groups based on specified conditions.

5. In paragraph 3 or 4, The above instructions, when executed by the at least one processor, cause the server to: A server set to determine the plurality of experimental groups, including a first experimental group confirmed in a first group among the plurality of groups and a second experimental group confirmed in a second group.

6. In any one of the clauses 3 to 5, The above instructions, when executed by the at least one processor, cause the server to: A server set to determine the plurality of experimental groups, including the first experimental group and the second experimental group confirmed in the first group among the plurality of groups.

7. A server in which the plurality of experimental models are set to differ in at least one of attributes, setting variables, functions, UI elements, statistical models, or algorithms in any one of the third to sixth clauses.

8. In any one of the clauses 3 to 7, The above instructions, when executed by the at least one processor, cause the server to: Confirming receipt of an application update request from a first electronic device among the plurality of electronic devices, A server configured to transmit a first experimental model set in the first experimental group among the plurality of experimental models to the first electronic device when it is confirmed that the first electronic device is included in a first experimental group among the plurality of experimental groups based on the identification information of the first electronic device.

9. In any one of the clauses 3 to 8, The above instructions, when executed by the at least one processor, cause the server to: A server configured to transmit a model set for the third group to the first electronic device when it is confirmed based on the identification information of the first electronic device that the first electronic device is included in a third group that is not included in the plurality of experimental groups among the plurality of groups, and to transmit a default model to the first electronic device when a model for the third group including the first electronic device is not set.

10. In any one of the clauses 3 to 9, The above instructions, when executed by the at least one processor, cause the server to: A server configured to provide a first user interface for displaying data comparing the plurality of experimental groups based on analysis of the user's usability data, and a second user interface for displaying statistical similarities between the plurality of experimental groups based on analysis of the user's characteristic data and the user's usability data.

11. In the method of operating an electronic device, An action of transmitting user characteristic data collected while multiple applications are running on the electronic device to a server; An operation of updating a first application among the plurality of applications that supports a control experiment using the received experimental model determined based on at least a portion of the user's characteristic data from the server; and A method comprising an action of transmitting user usage data collected while the above updated first application is running to the server.

12. In paragraph 11, A method further comprising transmitting the user's characteristic data, which includes first characteristic data collected while the first application is running and second characteristic data converted into keywords from a plurality of signals detected while the plurality of applications are running, to the server.

13. Regarding the server's operation method, An operation of determining a plurality of experimental groups among a plurality of electronic devices based on user characteristic data received from the plurality of electronic devices; An operation of determining a plurality of experimental models related to an application to be set for each of the plurality of experimental groups; An operation of transmitting a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups; An operation of transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups; and A method comprising an action of determining an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user usability data received from the plurality of experimental groups.

14. In a non-volatile storage medium storing commands, the commands are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation is: An action of transmitting user characteristic data collected while multiple applications are running on the electronic device to a server; An operation of updating a first application among the plurality of applications that supports a control experiment using the received experimental model determined based on at least a portion of the user's characteristic data from the server; and A storage medium including an operation for transmitting user usage data collected while the above updated first application is running to the server.

15. In a non-volatile storage medium storing commands, the commands are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation is: An operation of determining a plurality of experimental groups among a plurality of electronic devices based on user characteristic data received from the plurality of electronic devices; An operation of determining a plurality of experimental models related to an application to be set for each of the plurality of experimental groups; An operation of transmitting a first experimental model among the plurality of experimental models to a plurality of first electronic devices included in a first experimental group among the plurality of experimental groups; An operation of transmitting a second experimental model among the plurality of experimental models to a plurality of second electronic devices included in a second experimental group among the plurality of experimental groups; and A storage medium including an operation of determining an experimental model for the plurality of experimental groups among the plurality of experimental models based on the user usability data received from the plurality of experimental groups.

Images