Pre-commit test run minimization analysis using runtime isolation assurance

Abstract

Pre-commit testing is employed in a manner that can significantly reduce processing resources and time for verifying code modules. In one aspect, this includes: identifying a component to be subjected to a pre-commit test (702); building a set of tests for the component (704); analyzing the set of tests to identify any tests of the seal (706); separating the identified seal tests from any non-seal tests, including identifying whether any of the seal tests will be affected by the current change (708); and performing a pre-commit test on all of the non-seal tests and only seal tests that will be affected by the current change (710). For any of the components that pass the pre-commit test, those components may be stored in a code repository (712). Thus, if the sealed test packet is determined not to be affected by a given change, that test will be skipped in the pre-commit test.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Patent Application No. 17 / 973,647, filed on October 26, 2022, the entire disclosure of which is expressly incorporated herein by reference. Background Art

[0002] A software project may include any number of modules created by one or more developers. These modules may each be written in a variety of programming languages. Before adding a new or updated module to a software build, testing is typically performed to determine if there are potential issues with the module. This may include assessing whether the new functionality works as intended, whether modifications destroy existing functionality, whether there is an impact on performance, etc.

[0003] One or more forms of validation testing may be performed to ensure that various requirements are met. At one level, this may include unit testing of individual software components. At another level, regression testing may be used to ensure that new or updated code does not inadvertently affect existing features. For large projects where there are large code bases of hundreds or thousands (or more) modules, such testing may be resource intensive. This may involve a large amount of computing resources (e.g., multiple servers to run various tests) and may take hours or longer. Summary of the invention

[0004] Aspects of the present technology provide a software testing framework that can significantly reduce the processing resources and time required for verifying code modules. This can include using a sealed evaluator tool that is configured to parse software components into one or more sealed test packages. Tests that can pass in a sealed environment can then be run in a sealed environment, but this may only be done when a change list modifies the test. If a sealed test package is determined to be unaffected by a given change, that test will be skipped in pre-submission testing. This approach can significantly reduce pre-submission latency and overall computing resources. It can also save inelastic resources, such as laboratory hardware inventory for tests that cannot be run on emulators.

[0005] According to one aspect, a pre-submission testing method includes: identifying, by one or more processors of a computing system, components to be subjected to pre-submission testing; constructing, by one or more processors, a set of tests for the components; analyzing, by one or more processors, the set of tests to identify any tests that are sealed; separating, by one or more processors, the identified sealed tests from any non-sealed tests, including identifying whether any sealed tests in the sealed tests will be affected by a current change; performing, by one or more processors, pre-submission testing on all non-sealed tests in the non-sealed tests and only on sealed tests that will be affected by the current change; and, for any components that pass the pre-submission testing, storing those components in a code repository.

[0006] Analyzing the set of tests to identify any test that is sealed may include checking an indicator for the test package to confirm whether the test package is sealed. The indicator may be associated with the component manifest file. Here, the indicator may identify a given test as sealed if and only if the given test has no dependencies of an absolute uniform resource locator (URL) and is not associated with any external component.

[0007] As an alternative or in addition to the above, analyzing the set of tests to identify any tests that are sealed can include determining that a given test package will not be affected by a particular change. As an alternative or in addition to the above, analyzing the set of tests to identify any tests that are sealed can include determining that a given component will not be affected by a particular change. As an alternative or in addition to the above, performing pre-submission testing on all non-sealed tests in the non-sealed tests can include performing one or more unit tests on at least a subset of the non-sealed tests. As an alternative or in addition to the above, identifying the components to be subjected to pre-submission testing can be based on receiving notification of the current change.

[0008] As an alternative or supplement to the above, analyzing the group of tests may include applying a sealed package parser that forces confirmation that a given test component in the test component is independent of any component outside the test package. Here, the sealed package parser may define how to convert a component startup uniform resource locator (URL) into a component declaration and namespace specification at runtime. The sealed package parser may be configured to force confirmation that all resolution requests are resolved within the test package. In this case, when all resolution requests cannot be resolved within the test package, the method may include the sealed package parser returning an error to the system package parser.

[0009] According to another aspect, a system is provided, the system including a code repository and a computing system including one or more processors operatively coupled to the code repository. The one or more processors are configured to: identify components to be subject to pre-submission testing; build a set of tests for the components; analyze the set of tests to identify any tests that are sealed; separate the identified sealed tests from any non-sealed tests, including identifying whether any sealed tests in the sealed tests will be affected by the current change; perform pre-submission testing on all non-sealed tests in the non-sealed tests and only on the sealed tests that will be affected by the current change; and for any components that pass the pre-submission testing, store those components in the code repository.

[0010] According to another aspect, a pre-submit testing method includes: identifying, by one or more processors of a computing system, a component to be subjected to pre-submit testing, wherein the pre-submit testing includes executing a set of tests; analyzing, by the one or more processors, changes to the component to be subjected to pre-submit testing since a previous execution of the pre-submit testing; identifying, by the one or more processors, one or more sealing tests from the set of tests, any output of the one or more sealing tests being individually defined by one or more associated sealing test packages among the component; executing, by the one or more processors, the pre-submit testing, wherein the pre-submit testing includes executing the one or more sealing tests only when it is determined that one or more of the changes to the component will affect the one or more sealing tests; and storing, for any component that passes the pre-submit testing, in a code repository.

[0011] Analyzing changes to components that are to be subject to pre-submission testing may include applying a sealed package parser that enforces confirmation that a given component in the component is not dependent on any component outside the test package. As an alternative or in addition to the above, applying the sealed package parser may include evaluating a corresponding component manifest file for a given component. Here, evaluating the corresponding component manifest file may include evaluating whether a sealing key associated with a given manifest file satisfies a sealing condition. As an alternative or in addition to the above, performing pre-submission testing may include performing one or more unit tests. As an alternative or in addition to the above, identifying components that are to be subject to pre-submission testing may be based on receiving notification of changes to the components.

[0012] According to another aspect, a system is provided, the system comprising a code repository and a computing system, the computing system comprising one or more processors operatively coupled to the code repository. The one or more processors are configured to: identify a component to be subjected to pre-submission testing, wherein the pre-submission testing comprises executing a set of tests; analyze changes to the component to be subjected to pre-submission testing since a previous execution of the pre-submission testing; identify one or more sealing tests from the set of tests, any output of the one or more sealing tests being individually defined by one or more associated sealing test packages among the components; perform the pre-submission testing, wherein the pre-submission testing comprises performing the one or more sealing tests only when it is determined that one or more of the changes to the component will affect the one or more sealing tests; and for any of the components that pass the pre-submission testing, store it in the code repository. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a development diagram based on various aspects of this technology.

[0014] Figure 2 is a flowchart of various aspects of the present technology.

[0015] FIG. 3A to FIG. 3C Examples of component testing in accordance with aspects of the present technique are shown.

[0016] Figure 4 Flowcharts in accordance with aspects of the present technique are shown.

[0017] Figure 5 A block diagram is shown of an example computing device that may be employed in accordance with aspects of the present technique.

[0018] FIG. 6A to FIG. 6B A system for use with various aspects of the present technology is shown.

[0019] Figure 7 Methods according to aspects of the present technology are shown.

[0020] Figure 8 Another method in accordance with aspects of the present technique is shown. DETAILED DESCRIPTION Overview

[0021] A pre-commit testing framework can be used to ensure that a project's code passes all necessary tests before being added to a code repository. A code repository can be a global repository accessible to all developers in a given team or company. Figure 1An example scenario 100 is shown in which multiple developers 102 are each responsible for a different software component. A software component may be code for an application, a native application, and / or system software to be executed by an operating system, etc. Once created, the software components may all be stored as part of a code repository in a memory 104, such as in a code repository 106. The components may be accessible to the developers 102 and other users 108 of the code repository 106.

[0022] In one scenario, a company or development team can adopt a continuous integration approach, in which any code update can cause testing to be executed to identify regressions early in the software product development process. Certain tests, particularly pre-submission tests, can be run before a change list (CL) is submitted to a code repository. For example, subtrees in a code repository can each contain configuration files that determine which tests to run and when they should run (e.g., during code review, immediately before submitting a CL, etc.). Pre-submission testing can be run synchronously before sending changes for review or before submitting a CL to a repository. It can also be run asynchronously as needed. Post-submission testing can be run after a CL has been submitted. If the pre-submission test fails, the CL may not be submitted to the repository. Post-submission testing can be run in the background and is usually not time-sensitive.

[0023] In some cases, a distributed build system can be used to compile and link software for running tests. This can include providing a set of common commands for building and testing code and executing tests. The computing system can distribute the work of each build across many computing devices, such as hundreds or thousands of cores or servers. According to one aspect, individual build steps can be both sealed and deterministic. Sealed build steps rely only on declared inputs (including compilers called by the build system), which enables the build system to know all the real dependencies of the component. This sealing can ensure that the build steps are deterministic.

[0024] Figure 2 An example test workflow 200 is shown in which a developer 202 generates a test report to be submitted to Figure 1 206. Before a CL is submitted to the repository, pre-submission testing occurs at block 204. This may include unit testing. The results of the pre-submission testing may be reviewed by a developer 202. The results may also be cross-checked or otherwise evaluated by another developer 206. For example, a developer 206 may request that any changes that add new functionality be covered by the test.

[0025] The pre-submission testing process may apply the new or updated code to the corresponding portion of the system software code and generate a binary of the resulting code. If the resulting code does not pass the build test, a build error is generated. Pre-submission testing may include testing to identify memory leaks, use performance profiling, perform code coverage testing, buffer overflow testing, etc. Once any issues with the software component have been resolved, the component may be stored in the repository 106. Subsequently, post-submission testing 208 may be performed. Then, at block 210, once testing has been completed, a product build may be created. Example Method

[0026] Pre-commit testing, whether it includes unit tests and / or other types of tests, typically requires significant computing resources and time to complete. Continuous integration approaches may involve more pre-commit testing than other approaches, which can be magnified when the code base reaches hundreds or thousands of modules or more. Therefore, ways to reduce the amount of time and / or resources required to test the system provide direct tangible benefits.

[0027] In some cases, machine learning, regression analysis, or other statistical techniques can be used to estimate or guess which tests can be skipped. However, the consequences of not performing a test—which may result in a false negative (which assumes that the test will not fail)—can be severe depending on the type of failure that may have occurred and should have been discovered in pre-submission testing.

[0028] The pre-submission test framework provided herein relates to analyzing new or updated code to detect which test packages will be affected by changes. The system can then perform analysis, wherein a group of unaffected test packages and the group of sealed tests are intersected. This intersecting identification will not be affected by changes. In this way, the process can exclude those sealed tests from pre-submission testing. Creating a test package that omits unaffected sealed tests can significantly reduce the amount of pre-submission testing that originally needed to run. For example, according to some experiments, the pre-submission test load on the test system can be reduced by about 60% or more, and in some cases, the test load reduction may be as much as 90%. This significantly reduces the amount of processing resources originally devoted to pre-submission testing. In addition, the reduction of the amount of testing will inevitably lead to the reduction of the amount of time (which can be regarded as delay) required for executing pre-submission testing. Pre-submission testing

[0029] Depending on the underlying operating system architecture, the software platform may or may not incorporate runtime isolation mechanisms. For example, when runtime isolation is employed, processes without any capabilities may be created by default, which enables the component framework to fully control the sandbox of any component to be executed by the system. In one scenario, software may be distributed via immutable, cryptographically encapsulated sealed packages. Such packages may each include a hierarchical collection of files that provide a set of programs, components, or services that can be run on a device. A package may represent a distribution unit composed of parts that may not be a single binary.

[0030] Components are considered isolated from each other if they cannot affect the behavior of other components with their own behavior or existence. A test package is considered sealed if its behavior is completely defined by the content to be distributed, such as the files in the test package, for example, the test program and any other assets packaged with the test program, such as configuration files or golden files. A test component can be considered sealed when the unit test does not depend on any other components or resources. A test component can also be considered sealed when it depends only on other components packaged in the same test package (sealed test package).

[0031] When two or more tests are isolated from each other, they can be run in any order or in parallel. The system can manipulate the order to simplify the slicing of the test workload given the available computing resources. Executing isolated tests in parallel can reduce testing latency. In addition, tests that are not changed by other isolated tests can be skipped during testing. This can be particularly beneficial because it can potentially reduce the test load required to verify incoming changes by several orders of magnitude.

[0032] In some cases, a component and its child (sub)components may reside in the same package or in different packages. When dependencies reside in the same package as the component, relative uniform resource locators (URLs) may be used to resolve the component topology. However, when the parent and child components are in separate packages but packaged in a sealed test package for testing purposes, a sealed evaluator tool such as an isolated sealed (package) resolver may be employed. A sealed resolver may be implemented as a service that defines how to convert a component launch URL into a component declaration and sandbox / namespace specification at runtime.

[0033] Sealed parsers can be used to identify which tests should be considered completely sealed at build time. Sealed parsers enforce confirmation or otherwise ensure that test components do not depend on components outside the test package. Then, at runtime, the system can send selected tests to an execution environment that enforces confirmation of isolation and sealing. Here, it is important to note that any test that was previously passed within a sealed environment should continue to pass as long as the test package to which it is applied remains unmodified, and its results can therefore be cached. Moreover, any test that can pass in a sealed environment should be run in a sealed environment. For any test that meets such criteria and may have been run previously, there is no need to rerun such test unless there is a change in the sealed test package to which the test is applied. According to one aspect of the present technology, this can significantly improve the efficiency and scalability of the submission queue. The amount of testing completed per CL becomes proportional to the number of build targets modified by changes rather than corresponding to the size of the code base.

[0034] Figure 3A An example scenario 300 is shown in which a component A.cm (where ".cm" indicates a component manifest) for a specific operating system (OS) has two subcomponents B.cm and C.cm. A component manifest is used to describe the structure of a specific component, for example, the type of runtime in which the component is executed, a list of all dependencies, and any rules for managing the resources owned and used by the component. The manifest can be compiled in a binary form using a .cm extension. As shown, each package has its own URL. Each subcomponent in a subcomponent is part of a different package that is different from the parent package. Ideally, the tests of A.cm, B.cm, and C.cm should be packaged in a test package using the manifest files of each component without any modification. Figure 3B A view 320 is shown showing how a test package 322 for A, B, and C may be constructed.

[0035] According to one aspect of the present technology, a sealed resolver will always ignore the package portion in the component url and resolve the cm file associated with the component to a predefined package. The component URL contains the name of the package that includes the component. For example, in "OS-pkg: / / OS.com / test_package#metaA / tests.cm", the name of the package containing the desired component is "test_package". A sealed resolver can be injected when executing a test component so that all resolution requests from that test component are provided by the isolated sealed resolver.

[0036] In one scenario, a sealed resolver can be a local component added to a subdomain of each test, where the scope of component resolution is limited to a discrete set of packages, including the test package, subpackages of the test package, and a set of allow-listed packages for resolving external package resolution (which can be used to establish regression stops). In an example, the allow list can be statically coded, but the test packages and subpackages can be dynamically changed for each test package to be run.

[0037] The sealed resolver parses this out of all requests to enforce that all resolution requests are resolved within packages. Specifically, if the resolver receives a request from some arbitrary package 'A', it ensures that any resolution component's request is scoped to: package A, subpackages of package A, and other allowed list packages. These 3 groups of packages form the set of allowed packages. Figure 3C A flow 340 is shown indicating how a sealed parser may operate. At box 342, the sealed parser parses package A from the url. At box 344, it is determined that package A is inside the set of allowed packages. If so, then at box 346, the sealed parser forwards the parsing of the package to the system package parser. However, when it is determined at box 348 that package A is not inside the set of allowed packages, an error will be returned at 350. The system package parser is configured to perform parsing across the entire system, so the sealed parser can be viewed as a filter on top of the system package parser.

[0038] According to another aspect of the present technology, a label, flag, mark or other indicator can be associated with each test, depending on what is packaged and whether it is sealed. For example, a test type indicator can identify a test as being sealed or unsealed. Alternatively, the label can use a generated component manifest (.cm file) as a source of truth as to whether the test or the component associated with the test is sealed. In one scenario, the sealing key associated with the manifest may be true or false. In this scenario, the sealing key will be true if and only if the test is executed and packaged in a sealed manner, has no dependencies on absolute URLs, and has no external capabilities.

[0039] Figure 4 An example pre-submission process flow 400 is shown. At box 402, one or more components are generated by a developer. At box 404, the system builds a set of tests to evaluate the components. This can include unit tests and / or other types of tests. The slicing of the tests can include selecting the order of the tests, which machines will process certain tests, etc. Once the set of tests has been built, the computing system analyzes the set at box 406 to identify any sealed tests. This can include identifying whether a given sealed test has been previously run in an earlier pre-submission testing process. Therefore, the system filters out any sealed tests that are not affected by the current changes. The system can discard any previously performed sealed tests because their results will not change.

[0040] At box 408, the system then performs pre-submission testing on all non-sealed tests and any remaining sealed tests that are considered to be affected by the change list. Since sealed tests have been marked or otherwise excluded, there may be a significant reduction in the computing resources used to perform pre-submission testing (e.g., the reduction may be about 60% or more) and / or a significant reduction in the time required. Then, at box 410, the system evaluates whether each such test has passed. For example, as described above, pre-submission testing may include tests that identify memory leaks, use performance profiling, perform code coverage testing, buffer overflow testing, etc. If a given test passes, the component of that test is stored in the code repository, as shown in box 412. However, if a given test fails for some reason, at box 414, additional evaluation of the component can be performed automatically by the developer who created the component, another developer, or by the system. Once the components have been stored in the code repository, they can be implemented in a build package, such as an update for an operating system, a feature upgrade, or a new app. The build package can be uploaded to one or more computing devices as needed or upon request.

[0041] As described above, some software platforms may incorporate runtime isolation mechanisms. This may provide enhanced overall security for the platform, for example because processes without any capabilities may be created by default, which enables the component framework to fully control the sandbox of any component to be executed by the system. Other platforms may not specifically implement runtime isolation, or may not require such isolation at all. In such cases, it may be helpful to include certain requirements to ensure that certain tests are actually sealed. For example, component packages may be fully defined in terms of content using Merkle hashes or other standards. Regardless of which method the software platform utilizes, the pre-submission testing method discussed above may be performed as long as the method enables the build system to know all the real dependencies of the component. Example computing device

[0042] The present technology can be applied in software assembly to applications that can be used with many different types of computing devices. This can include desktop computing devices, laptop computing devices (e.g., netbooks, tablets, etc.), mobile phones, interactive home appliances, smart televisions, etc. Once the components are stored in the code repository, they can be provided to computing devices, such as part of an operating system update, feature upgrade, etc.

[0043] Figure 5A block diagram 500 of an example computing device such as a desktop device, a laptop device, or an interactive home appliance type device is shown. As shown, the computing device includes: a processing module 502 having one or more computer processors, such as a central processing unit 504 and / or a graphics processor 506; and a memory module 508, which is configured to store instructions 510 and data 512, such as the components discussed above. The processors may or may not operate in parallel, and may include ASICs, controllers, and other types of hardware circuit systems. The processor is configured to receive information from a user through a user interface module 514, and present information to the user on a display device of a display module 516 via the user interface module. The display module 516 has a display interface, and may be configured as a touch screen that implements user input via a stylus or other tool or by a user physically touching the screen. Alternatively or additionally, contactless gesture input and / or audio input may be supported.

[0044] The user interface module 514 is configured to receive user input. The user interface module 514 can receive commands from the user via user input, and the commands are converted to submit to a given processor. The user interface module can be linked to a network browser (not shown). The user input can include a touch screen as described above as a supplement or replacement of a keyboard, a keypad, a mouse pad and / or a touchpad, a microphone, an input based on gestures or other types of input devices. The keyboard, keypad, mouse pad and / or touchpad can be a part of a computing device or can be connected to a computing device via a cable or other wired connection, or can be physically separated from the integrated client device and configured to be connected via one or more wireless connections such as Bluetooth , WiFi, ultra-wideband (UWB), infrared rays, etc. The user interface module 514 can be operably connected to the display module 516.

[0045] The display module 516 may include circuitry for driving a display device to present graphics and other information to a user. In other words, the display device is configured to present visual content. For example, graphics information may be generated by a graphics processor 506, while a central processing unit (CPU) 504 manages the overall operation of the computing device. The graphics information may be displayed on the display module 516 in response to a user query. For example, the processing module may use instructions and data stored in the memory module 508 to run a browser application, a game application, an enterprise app, or other service, and present information to the user via the display module 516. The memory module 508 may include a database or other storage for browser information, game state information, location information, etc.

[0046] The memory module 508 may be implemented as one or more of: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. The memory module 508 may include, for example, unmanaged fast memory and / or NVRAM (which may be NAND-based memory), and may be embodied as a hard drive or a memory card, such as an embedded multimedia card (eMMC) or a solid-state drive (SSD) card (e.g., "managed NAND" or "managed memory"). Alternatively, the memory module 508 may also include removable media (e.g., a DVD, CD-ROM, or USB thumb drive). According to one aspect, the memory module 508 may be configured to have multiple partitions.

[0047] One or more areas of memory module 508 may be writable, while other areas may include read-only (or otherwise write-protected) memory. In one implementation, a computer program product is tangibly embodied in an information carrier. Figure 5 The processor, memory module, and other elements of the integrated client device are functionally shown as being within the same overall box, but such components may or may not be stored within the same physical housing. For example, some or all of the instructions and data may be stored on an information carrier that is a removable storage medium (e.g., an optical drive, a high-density tape drive, or a USB drive) that is connectable to the base or display housing, and other instructions and data are stored in a read-only computer chip integrated into the base or display housing.

[0048] Data 512 can be retrieved, stored or modified by the processor according to instructions 510. For example, data can be stored in a computing device register, in a relational database as a table with multiple different fields and records, in an XML document or a flat file. Data can also be formatted in any computing device readable format. Instructions 510 can be any instruction set (such as machine code) that will be directly executed by the processor or any instruction set (such as a script) that is indirectly executed. For example, instructions can be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions" and "programs" can be used interchangeably herein. Instructions can be stored in an object code format for direct processing by the processor, or in any other computing device language, which includes scripts or independent source code module sets that are interpreted on demand or compiled in advance.

[0049] Also like Figure 5As shown in example 500 of , the computing device includes a communication module 518 for communicating with other devices and systems including other computing devices (e.g., a user's mobile phone or wearable computing device), servers, and databases. The communication module 518 includes a wireless transceiver; alternatively, the module may alternatively or additionally include a wired transceiver. The computing device can communicate with other remote devices via the communication module 518 using various configurations and protocols, including short-range communication protocols such as near field communication (NFC), Bluetooth™, Bluetooth™ Low Energy (BLE), UWB or other ad hoc networks, the Internet, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and combinations of the foregoing.

[0050] In addition, the example device as shown includes one or more position and orientation sensors 520. The position and orientation sensors 520 are configured to determine the position and orientation of one or more parts of the computing device, such as the display module relative to the base. For example, these components may include a GPS receiver for estimating the latitude, longitude and / or altitude of the integrated client device, and an accelerometer, gyroscope, or another direction / speed detection device capable of determining the orientation of the display housing relative to the base (as well as the rate of change of the positioning of the display housing), such as an inertial measurement unit (IMU).

[0051] The computing device may also include one or more cameras 522 for capturing still images and recording video streams, such as an integrated webcam and / or a dedicated imaging device for presence sensing as discussed above. The device may also include one or more microphones 523 (which may be used for command input and / or presence sensing, e.g., by detecting acoustic information within a threshold distance from the client device), a speaker 524, and a power module 526. An actuator for providing tactile feedback or other information to the user may be incorporated into a touch screen of a display module (not shown). Example Network

[0052] As described above, various types of products can be updated in the described manner with components that have undergone pre-submission testing. In different scenarios, updates can be performed on all devices in a product line, a subset (batch) of devices in a product line, multiple different product lines, etc. Alternatively, a specific app or feature can be added to a product by downloading it from a code repository or other storage system. Fig. 6A and Figure 6B An example computing architecture that can be employed in these methods is shown in FIG. In particular, Fig. 6A and Figure 6B606. FIG. 607 is a diagram and a functional diagram of an example system 600, which includes multiple computing devices and databases connected via a network. For example, computing device 602 can be a cloud-based server system that provides or otherwise supports updates for various products. Database 604 can store updates and other information associated with products and / or platforms. In one example, a given database can include a code repository for a company or development team. The server system can access the database via network 606.

[0053] The computing device 608 may be a workstation or other computing device that may be used by a developer to create various components that may be added to a code repository. Products that may receive components may include one or more of the following: a desktop computer 610a, a laptop or tablet PC 610b, a home device that may include a portable unit (such as a home assistant device 612a or a smart speaker 612b) or a fixed unit (such as a temperature / thermostat unit 612c). Other products may include a personal communication device such as a mobile phone or PDA 614 or a wearable device such as a smart watch 616, etc.

[0054] In one example, computing device 602 may include one or more server computing devices having multiple computing devices, such as a load balancing server farm or a cloud computing system, which exchange information with different nodes of a network for the purpose of receiving, processing, and transmitting data to and from other computing devices. For example, computing device 602 may include one or more server computing devices that are configured to perform pre-submission testing and are capable of communicating with development device 608 and any of products 610 to 616 via network 606.

[0055] like Figure 6BAs shown, each of the computing device 602, the developer device 608, and the products 610 to 616 may include one or more processors, memory, data, and instructions. The memory stores information accessible by one or more processors, including instructions and data that can be executed or otherwise used by the processor. The memory can be any type capable of storing information accessible by the processor, including computing device readable media. The memory is a non-temporary medium such as a hard drive, a memory card, an optical disk, a solid state, etc. The system may include different combinations of the foregoing, thereby storing different parts of the instructions and data on different types of media. The instruction may be any instruction set (such as machine code) to be directly executed by the processor or any instruction set (such as a script) to be indirectly executed. For example, the instruction may be stored as a computing device code on a computing device readable medium. In this regard, the terms "instruction", "module" and "program" may be used interchangeably herein. The instruction may be stored in an object code format for direct processing by the processor, or stored in any other computing device language, which includes a script or independent source code module set that is interpreted or compiled in advance on demand.

[0056] The processor may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC, a graphics processing unit (GPU), a tensor processing unit (TPU), or other hardware-based processor. Figure 6B The processor, memory, and other elements of a given computing device are functionally shown as being located within the same box, but such a device may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage medium located in a housing different from that of the processor (e.g., located in a cloud computing system such as server 602). Therefore, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0057] Developer device 608 and products 610 to 616 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, as well as a user interface subsystem for receiving input from a user and presenting information (e.g., text, images, and / or other graphical elements) to the user. The user interface subsystem may include one or more user inputs (e.g., at least one front (user) camera, mouse, keyboard, touch screen, and / or microphone) and one or more display devices operable to display information (e.g., text, images, and / or other graphical elements). Other output devices such as speakers may also provide information to the user.

[0058] Development device 608 and / or products 610 to 616 may communicate with a backend computing system (e.g., server 602) via one or more networks such as network 606. Network 606 and intermediary nodes may include various configurations and protocols including short-range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing. Such communications may be facilitated by any means of transmitting data to and from other computing devices such as modems and wireless interfaces. Exemplary Operation Methods

[0059] Figure 7 An example pre-submission testing method 700 is shown. At box 702, one or more processors of a computing system identify a component to be subjected to pre-submission testing. At box 704, the method includes building a set of tests for the component by one or more processors. At box 706, the method includes analyzing the set of tests by one or more processors to identify any test that is sealed. At box 708, the method includes separating the identified sealed tests from any non-sealed tests by one or more processors, including identifying whether any sealed tests in the sealed tests will be affected by the current change. At box 710, the method includes performing pre-submission testing on all non-sealed tests in the non-sealed tests and only on the sealed tests that will be affected by the current change by one or more processors. And at box 712, for any component that passes the pre-submission test, the method includes storing those components in a code repository.

[0060] Figure 8 A pre-submission test method 800 is shown. At box 802, the method includes identifying, by one or more processors of a computing system, a component to be subjected to pre-submission testing, wherein the pre-submission testing includes executing a set of tests. At box 804, the method includes analyzing, by one or more processors, changes to the component to be subjected to pre-submission testing since the previous execution of the pre-submission testing. At box 806, the method includes identifying, by one or more processors, one or more sealing tests from the set of tests. Any output of one or more sealing tests is defined individually by one or more associated sealing test packages among the components. At box 808, the method includes executing the pre-submission test by one or more processors. This includes executing one or more sealing tests only when it is determined that one or more changes in the changes to the component will affect one or more sealing tests. And at box 810, for any component in the component that passes the pre-submission test, the method includes storing it in a code repository.

[0061] Although the technology herein has been described with reference to specific embodiments, it should be understood that these embodiments are only illustrative of the principles and applications of the present technology. Therefore, it should be understood that numerous modifications may be made to the illustrative embodiments, and other arrangements may be designed without departing from the spirit and scope of the present technology as defined by the appended claims.

Claims

1. A pre-submission testing method, include: identifying, by one or more processors of a computing system, components to be subjected to pre-submission testing; constructing, by the one or more processors, a set of tests for the component; analyzing, by the one or more processors, the set of tests to identify any tests that are sealed; separating, by the one or more processors, the identified sealing tests from any non-sealing tests, including identifying whether any of the sealing tests will be affected by the current change; performing, by the one or more processors, a pre-submission test on all of the non-sealed tests and only the sealed tests that will be affected by the current change; as well as For any of the components that pass the pre-submission test, those components are stored in a code repository. 2 . The method of claim 1 , wherein analyzing the set of tests to identify any test that is sealed comprises checking an indicator for a test packet to confirm whether the test packet is sealed. The method of claim 2 , wherein the indicator is associated with a component manifest file.

4. The method of claim 2, wherein the indicator identifies a given test as sealed if and only if the test has no dependencies of absolute uniform resource locators (URLs) and is not associated with any external components.

5. The method of claim 1, wherein analyzing the set of tests to identify any tests that are sealed comprises determining that a given test package will not be affected by a particular change.

6. The method of claim 1, wherein analyzing the set of tests to identify any test that is sealed includes determining that a given component will not be affected by a particular change. 7 . The method of claim 1 , wherein performing the pre-submission testing on all of the non-hermetic tests comprises performing one or more unit tests on at least a subset of the non-hermetic tests.

8. The method of claim 1, wherein identifying the components to be subject to pre-submission testing is based on receiving notification of the current change.

9. The method of claim 1, wherein analyzing the set of tests comprises applying a sealed package parser that enforces confirmation that a given one of the test components does not depend on any component external to the test package.

10. The method of claim 9, wherein the sealed package resolver defines how to convert a component launch uniform resource locator URL into a component declaration and namespace specification at runtime.

11. The method of claim 9, wherein the sealed packet parser is configured to forcibly confirm whether all parsing requests are parsed within the test packet.

12. The method of claim 11, wherein when all parsing requests cannot be parsed within the test packet, the method comprises the sealed packet parser returning an error to the system packet parser.

13. A pre-submission testing method, include: identifying, by one or more processors of a computing system, components to be subjected to pre-submission testing, wherein the pre-submission testing includes executing a set of tests; analyzing, by the one or more processors, changes to the component to be subject to pre-submission testing since a previous execution of the pre-submission testing; identifying, by the one or more processors, one or more sealing tests from the set of tests, any outputs of the one or more sealing tests being individually defined by one or more associated sealing test packages among the components; performing, by the one or more processors, the pre-submission testing, wherein the pre-submission testing includes performing the one or more sealing tests only if it is determined that one or more of the changes to the component will affect the one or more sealing tests; as well as For any of the components that pass the pre-submission test, stored in a code repository.

14. The method of claim 13, wherein analyzing the changes to the components subject to pre-submission testing comprises applying a sealed package parser that enforces confirmation that a given one of the components does not depend on any component outside of a test package.

15. The method of claim 13, wherein applying the sealed package parser comprises evaluating a corresponding component manifest file for the given component.

16. The method of claim 15, wherein evaluating the corresponding component manifest file comprises evaluating whether a sealing key associated with a given manifest file satisfies a sealing condition. The method of claim 13 , wherein performing the pre-submission testing comprises performing one or more unit tests.

18. The method of claim 13, wherein identifying the component to be subject to pre-submission testing is based on receiving notification of the change to the component.

19. A system, include: Code repository; as well as A computing system comprising one or more processors operatively coupled to the code repository, the one or more processors configured to: Identify components to be subjected to pre-submission testing; Building a set of tests for said components; analyzing the set of tests to identify any tests that are sealed; Separating the identified sealing tests from any non-sealing tests, including identifying whether any of the sealing tests will be affected by the current change; performing pre-submission testing on all of the non-sealed tests and only on the sealed tests that will be affected by the current change; and For any of the components that pass the pre-submission test, those components are stored in the code repository.

20. A system, include: Code repository; as well as A computing system comprising one or more processors operatively coupled to the code repository, the one or more processors configured to: identifying components to be subjected to pre-submission testing, wherein the pre-submission testing includes executing a set of tests; analyzing changes to the component to be subjected to pre-submission testing since a previous execution of the pre-submission testing; identifying one or more sealing tests from the set of tests, any outputs of the one or more sealing tests being individually defined by one or more associated sealing test packages among the components; performing the pre-submission testing, wherein the pre-submission testing includes performing the one or more sealing tests only if it is determined that one or more of the changes to the component will affect the one or more sealing tests; as well as For any of the components that pass the pre-submission test, stored in the code repository.

Images